Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Centralizing logs generated in EC2 instances.


Copy link to this message
-
Re: Centralizing logs generated in EC2 instances.
Well if tail -f does not work for you then you may look at writing logs
directly into the flume channel via piped logs.

Another way is you sink your data at multiple places and then may be if you
got enough computation power you can go for dedup logic or you have a size
check on the original file and sinked file

Spooldirectory may work for you if you add logrotation to your system.

(From the time I have used flume, I have not faced problems with tail -F)
On Wed, Dec 26, 2012 at 7:07 PM, Shankar <[EMAIL PROTECTED]> wrote:

> Nitin,
>
> Thanks for the quick reply.
>
> I went through this<http://flume.apache.org/FlumeUserGuide.html#flume-sources>.
> Not able to decide which Flume Source should I use.
> --> It doesn't seem to recommend using Exec Source as the event may get
> lost and hence no guarantee of delivery.
> --> Spooling dir source won't work for me as it works only for immutable
> files.
>
> Reliable delivery is very important for me.
>
> Can you suggest Flume Source which will work for me?
>
> On Wed, Dec 26, 2012 at 6:58 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
>> yes flume will definitely solve this problem
>>
>> look at flume's source
>> there are multiple ways you can feed the logs to flume .. simplest would
>> be tail -f in the flume exec source
>>
>>
>> On Wed, Dec 26, 2012 at 6:37 PM, Shankar <[EMAIL PROTECTED]> wrote:
>>
>>> I have an application running on multiple EC2 instances. I need to
>>> aggregate the logs generated in those instances to a central location.
>>> i.e., the logs generated on the EC2 instances automatically become
>>> available on the centralized server. I have logs generated by my
>>> application and many 3rd part applications, system logs such as syslog,
>>> secure log etc.
>>>
>>> I have following questions:
>>> 1. Can Flume solve this requirement?
>>> 2. Who is going to feed the log files to Flume agent? Do I need some
>>> other tool to feed my logs to Flume?
>>>
>>> --
>>> Regards,
>>> Varun Shankar
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> Regards,
> Varun Shankar
>

--
Nitin Pawar
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB