Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Best way to collect Hadoop logs across cluster


+
Mark Kerzner 2013-04-19, 04:23
+
Roman Shaposhnik 2013-04-19, 04:44
Copy link to this message
-
Re: Best way to collect Hadoop logs across cluster
Mark Kerzner 2013-04-26, 00:08
Thank you for all the advice, it was indeed very useful.

Mark
On Thu, Apr 18, 2013 at 11:44 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> On Thu, Apr 18, 2013 at 9:23 PM, Mark Kerzner <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > my clusters are on EC2, and they disappear after the cluster's instances
> are
> > destroyed. What is the best practice to collect the logs for later
> storage?
> >
> > EC2 does exactly that with their EMR, how do they do it?
>
> Apache Flume could be extremely useful for this purpose. You
> can even configure it to deposit log data in realtime into
> S3.
>
> Thanks,
> Roman.
>
+
Marcos Luis Ortiz Valmase... 2013-04-19, 04:51
+
Mark Kerzner 2013-04-19, 05:01
+
Marcos Luis Ortiz Valmase... 2013-04-19, 05:55