Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Help with spooling directory source


+
Robert George 2013-02-19, 11:08
+
Brock Noland 2013-02-19, 15:50
Copy link to this message
-
Re: Help with spooling directory source
Robert George 2013-02-20, 05:03
Thank you.
On Tue, Feb 19, 2013 at 9:20 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Hi,
>
> The spooling fir source expects immutable, uniquely named files
> as described here:
>
> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source
>
> As such you should log to a separate directory and then on roll move the
> file (uniquely named) into the spooling dir source.
>
> Brock
>
>
> On Tue, Feb 19, 2013 at 5:08 AM, Robert George <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I'm trying to use flumes Spooling Directory Source to move my logs from
>> ec2 instances to aws s3.
>>
>>
>> I'm using rotatelogs to create log files of 5MB sizes.
>>
>> I have one doubt. If I specify the spooldir sources directory as the
>> directory to which the apache creates log, will it work correctly.
>>
>> What I mean is, will flume wait till apache completes writing to the log
>> file?
>>
>> What I observe is, only couple of lines of the log file is coming in the
>> sink, but in the source side has more lines.
>>
>> I dont whether I put my question is correctly worder.
>>
>> My configruation files below.
>>
>> Apache configuration
>> -----------------------------
>>
>> CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
>> /mnt/je/logs/apache/jesites/epicenter/access/%Y-%m-%d-%H-%M-%S-access.log
>> 5M" cookie env=!dontlog
>> CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
>> /mnt/je/logs/apache/jesites/epicenter/access/%Y-%m-%d-%H-%M-%S-access_assets.log
>> 5M" combined env=dontlog
>> ErrorLog "|/opt/bitnami/apache2/bin/rotatelogs -l
>> /mnt/je/logs/apache/jesites/epicenter/error/%Y-%m-%d-%H-%M-%S-errorlog.log
>> 5M"
>>
>>
>> Flume configuration
>> ---------------------------
>> #source is of type spooling directory - epicenter access logs
>> agent1.sources.spooldir-epi-access.channels = ch1
>> agent1.sources.spooldir-epi-access.type = spooldir
>> agent1.sources.spooldir-epi-access.spoolDir >> /mnt/je/logs/apache/jesites/epicenter/access
>> agent1.sources.spooldir-epi-access.interceptors = i1 hostname type
>> agent1.sources.spooldir-epi-access.interceptors.i1.type = timestamp
>> agent1.sources.spooldir-epi-access.interceptors.hostname.type = host
>> agent1.sources.spooldir-epi-access.interceptors.hostname.useIP = false
>> agent1.sources.spooldir-epi-access.interceptors.hostname.preserveExisting
>> = true
>> agent1.sources.spooldir-epi-access.interceptors.type.type = static
>> agent1.sources.spooldir-epi-access.interceptors.type.key = type
>> agent1.sources.spooldir-epi-access.interceptors.type.value = epi-access
>> agent1.sources.spooldir-epi-access.fileHeader = true
>>
>>
>> --
>> Regards,
>>
>> Robert George
>>
>> http://justEat.in | [EMAIL PROTECTED] | +919986442677
>>
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
Regards,

Robert George

http://justEat.in | [EMAIL PROTECTED] | +919986442677