Actually the problem is not simple. Based on these problems, there are
three companies working on them:
Loggly is a part of the Amazon Marketplace:
How to do it:
- Splunk, with its Cloud platform Splunk Storm:
With any of these applications, you can collect permanently or based on
schedule all generated logs from Amazon EC2, which is the base service for
EMR and the others.
2013/4/19 Mark Kerzner <[EMAIL PROTECTED]>
> So you are saying, the problem is very simple. Just before you destroy the
> cluster, simply collect the logs to S3. Anyway, I only need them after I
> have completed with a specific computation. So I don't need any special
> In regular permanent clusters, is there something that allows you to see
> all logs in one place?
> Thank you,
> On Thu, Apr 18, 2013 at 11:51 PM, Marcos Luis Ortiz Valmaseda <
> [EMAIL PROTECTED]> wrote:
>> When you destroy an EC2 instance, the correct behavior is to erase all
>> Why don't you create a service to collect the logs directly to a S3
>> bucket in real-time or in a batch of 5 mins?
>> 2013/4/18 Mark Kerzner <[EMAIL PROTECTED]>
>>> my clusters are on EC2, and they disappear after the cluster's instances
>>> are destroyed. What is the best practice to collect the logs for later
>>> EC2 does exactly that with their EMR, how do they do it?
>>> Thank you,
>> Marcos Ortiz Valmaseda,
>> *Data-Driven Product Manager* at PDVSA
>> *Blog*: http://dataddict.wordpress.com/
>> *LinkedIn: *http://www.linkedin.com/in/marcosluis2186
>> *Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>
Marcos Ortiz Valmaseda,
*Data-Driven Product Manager* at PDVSA
*Twitter*: @marcosluis2186 <http://twitter.com/marcosluis2186>