Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Centralizing logs generated in EC2 instances.


+
Shankar 2012-12-26, 13:07
+
Nitin Pawar 2012-12-26, 13:28
Copy link to this message
-
Re: Centralizing logs generated in EC2 instances.
Nitin,

Thanks for the quick reply.

I went through this<http://flume.apache.org/FlumeUserGuide.html#flume-sources>.
Not able to decide which Flume Source should I use.
--> It doesn't seem to recommend using Exec Source as the event may get
lost and hence no guarantee of delivery.
--> Spooling dir source won't work for me as it works only for immutable
files.

Reliable delivery is very important for me.

Can you suggest Flume Source which will work for me?

On Wed, Dec 26, 2012 at 6:58 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> yes flume will definitely solve this problem
>
> look at flume's source
> there are multiple ways you can feed the logs to flume .. simplest would
> be tail -f in the flume exec source
>
>
> On Wed, Dec 26, 2012 at 6:37 PM, Shankar <[EMAIL PROTECTED]> wrote:
>
>> I have an application running on multiple EC2 instances. I need to
>> aggregate the logs generated in those instances to a central location.
>> i.e., the logs generated on the EC2 instances automatically become
>> available on the centralized server. I have logs generated by my
>> application and many 3rd part applications, system logs such as syslog,
>> secure log etc.
>>
>> I have following questions:
>> 1. Can Flume solve this requirement?
>> 2. Who is going to feed the log files to Flume agent? Do I need some
>> other tool to feed my logs to Flume?
>>
>> --
>> Regards,
>> Varun Shankar
>>
>
>
>
> --
> Nitin Pawar
>

--
Regards,
Varun Shankar
+
Nitin Pawar 2012-12-26, 14:38
+
Shankar 2012-12-26, 15:27