Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Help with spooling directory source

Copy link to this message
Help with spooling directory source
Robert George 2013-02-19, 11:08

I'm trying to use flumes Spooling Directory Source to move my logs from ec2
instances to aws s3.
I'm using rotatelogs to create log files of 5MB sizes.

I have one doubt. If I specify the spooldir sources directory as the
directory to which the apache creates log, will it work correctly.

What I mean is, will flume wait till apache completes writing to the log

What I observe is, only couple of lines of the log file is coming in the
sink, but in the source side has more lines.

I dont whether I put my question is correctly worder.

My configruation files below.

Apache configuration

CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
5M" cookie env=!dontlog
CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
5M" combined env=dontlog
ErrorLog "|/opt/bitnami/apache2/bin/rotatelogs -l
Flume configuration
#source is of type spooling directory - epicenter access logs
agent1.sources.spooldir-epi-access.channels = ch1
agent1.sources.spooldir-epi-access.type = spooldir
agent1.sources.spooldir-epi-access.spoolDir /mnt/je/logs/apache/jesites/epicenter/access
agent1.sources.spooldir-epi-access.interceptors = i1 hostname type
agent1.sources.spooldir-epi-access.interceptors.i1.type = timestamp
agent1.sources.spooldir-epi-access.interceptors.hostname.type = host
agent1.sources.spooldir-epi-access.interceptors.hostname.useIP = false
agent1.sources.spooldir-epi-access.interceptors.hostname.preserveExisting true
agent1.sources.spooldir-epi-access.interceptors.type.type = static
agent1.sources.spooldir-epi-access.interceptors.type.key = type
agent1.sources.spooldir-epi-access.interceptors.type.value = epi-access
agent1.sources.spooldir-epi-access.fileHeader = true

Robert George

http://justEat.in | [EMAIL PROTECTED] | +919986442677