-
Re: Centralizing logs generated in EC2 instances.Shankar 2012-12-26, 15:27
On Wed, Dec 26, 2012 at 8:08 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
> Well if tail -f does not work for you then you may look at writing logs
> directly into the flume channel via piped logs.
>
Is it more reliable than Exec Source with 'tail -F'? How does this work?
Do I have to modify my application to write to a pipe instead of a log
file? How do I ask a 3rd party app to do this (if it's not supported)? Any
tutorial/working example for this?
>
> Another way is you sink your data at multiple places and then may be if
> you got enough computation power you can go for dedup logic or you have a
> size check on the original file and sinked file
>
> Spooldirectory may work for you if you add logrotation to your system.
>
My log files are rotated once a day but I want my logs to be available on
the central server instantly.
>
> (From the time I have used flume, I have not faced problems with tail -F)
>
>
> On Wed, Dec 26, 2012 at 7:07 PM, Shankar <[EMAIL PROTECTED]> wrote:
>
>> Nitin,
>>
>> Thanks for the quick reply.
>>
>> I went through this<http://flume.apache.org/FlumeUserGuide.html#flume-sources>.
>> Not able to decide which Flume Source should I use.
>> --> It doesn't seem to recommend using Exec Source as the event may get
>> lost and hence no guarantee of delivery.
>> --> Spooling dir source won't work for me as it works only for immutable
>> files.
>>
>> Reliable delivery is very important for me.
>>
>> Can you suggest Flume Source which will work for me?
>>
>> On Wed, Dec 26, 2012 at 6:58 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>
>>> yes flume will definitely solve this problem
>>>
>>> look at flume's source
>>> there are multiple ways you can feed the logs to flume .. simplest would
>>> be tail -f in the flume exec source
>>>
>>>
>>> On Wed, Dec 26, 2012 at 6:37 PM, Shankar <[EMAIL PROTECTED]> wrote:
>>>
>>>> I have an application running on multiple EC2 instances. I need to
>>>> aggregate the logs generated in those instances to a central location.
>>>> i.e., the logs generated on the EC2 instances automatically become
>>>> available on the centralized server. I have logs generated by my
>>>> application and many 3rd part applications, system logs such as syslog,
>>>> secure log etc.
>>>>
>>>> I have following questions:
>>>> 1. Can Flume solve this requirement?
>>>> 2. Who is going to feed the log files to Flume agent? Do I need some
>>>> other tool to feed my logs to Flume?
>>>>
>>>> --
>>>> Regards,
>>>> Varun Shankar
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>> Regards,
>> Varun Shankar
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
Varun Shankar
> Well if tail -f does not work for you then you may look at writing logs
> directly into the flume channel via piped logs.
>
Is it more reliable than Exec Source with 'tail -F'? How does this work?
Do I have to modify my application to write to a pipe instead of a log
file? How do I ask a 3rd party app to do this (if it's not supported)? Any
tutorial/working example for this?
>
> Another way is you sink your data at multiple places and then may be if
> you got enough computation power you can go for dedup logic or you have a
> size check on the original file and sinked file
>
> Spooldirectory may work for you if you add logrotation to your system.
>
My log files are rotated once a day but I want my logs to be available on
the central server instantly.
>
> (From the time I have used flume, I have not faced problems with tail -F)
>
>
> On Wed, Dec 26, 2012 at 7:07 PM, Shankar <[EMAIL PROTECTED]> wrote:
>
>> Nitin,
>>
>> Thanks for the quick reply.
>>
>> I went through this<http://flume.apache.org/FlumeUserGuide.html#flume-sources>.
>> Not able to decide which Flume Source should I use.
>> --> It doesn't seem to recommend using Exec Source as the event may get
>> lost and hence no guarantee of delivery.
>> --> Spooling dir source won't work for me as it works only for immutable
>> files.
>>
>> Reliable delivery is very important for me.
>>
>> Can you suggest Flume Source which will work for me?
>>
>> On Wed, Dec 26, 2012 at 6:58 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>>
>>> yes flume will definitely solve this problem
>>>
>>> look at flume's source
>>> there are multiple ways you can feed the logs to flume .. simplest would
>>> be tail -f in the flume exec source
>>>
>>>
>>> On Wed, Dec 26, 2012 at 6:37 PM, Shankar <[EMAIL PROTECTED]> wrote:
>>>
>>>> I have an application running on multiple EC2 instances. I need to
>>>> aggregate the logs generated in those instances to a central location.
>>>> i.e., the logs generated on the EC2 instances automatically become
>>>> available on the centralized server. I have logs generated by my
>>>> application and many 3rd part applications, system logs such as syslog,
>>>> secure log etc.
>>>>
>>>> I have following questions:
>>>> 1. Can Flume solve this requirement?
>>>> 2. Who is going to feed the log files to Flume agent? Do I need some
>>>> other tool to feed my logs to Flume?
>>>>
>>>> --
>>>> Regards,
>>>> Varun Shankar
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>> Regards,
>> Varun Shankar
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
Varun Shankar
