Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Help with spooling directory source

Copy link to this message
Help with spooling directory source

I'm trying to use flumes Spooling Directory Source to move my logs from ec2
instances to aws s3.
I'm using rotatelogs to create log files of 5MB sizes.

I have one doubt. If I specify the spooldir sources directory as the
directory to which the apache creates log, will it work correctly.

What I mean is, will flume wait till apache completes writing to the log

What I observe is, only couple of lines of the log file is coming in the
sink, but in the source side has more lines.

I dont whether I put my question is correctly worder.

My configruation files below.

Apache configuration

CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
5M" cookie env=!dontlog
CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
5M" combined env=dontlog
ErrorLog "|/opt/bitnami/apache2/bin/rotatelogs -l
Flume configuration
#source is of type spooling directory - epicenter access logs
agent1.sources.spooldir-epi-access.channels = ch1
agent1.sources.spooldir-epi-access.type = spooldir
agent1.sources.spooldir-epi-access.spoolDir /mnt/je/logs/apache/jesites/epicenter/access
agent1.sources.spooldir-epi-access.interceptors = i1 hostname type
agent1.sources.spooldir-epi-access.interceptors.i1.type = timestamp
agent1.sources.spooldir-epi-access.interceptors.hostname.type = host
agent1.sources.spooldir-epi-access.interceptors.hostname.useIP = false
agent1.sources.spooldir-epi-access.interceptors.hostname.preserveExisting true
agent1.sources.spooldir-epi-access.interceptors.type.type = static
agent1.sources.spooldir-epi-access.interceptors.type.key = type
agent1.sources.spooldir-epi-access.interceptors.type.value = epi-access
agent1.sources.spooldir-epi-access.fileHeader = true

Robert George

http://justEat.in | [EMAIL PROTECTED] | +919986442677