Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Help with spooling directory source


Copy link to this message
-
Help with spooling directory source
Hi,

I'm trying to use flumes Spooling Directory Source to move my logs from ec2
instances to aws s3.
I'm using rotatelogs to create log files of 5MB sizes.

I have one doubt. If I specify the spooldir sources directory as the
directory to which the apache creates log, will it work correctly.

What I mean is, will flume wait till apache completes writing to the log
file?

What I observe is, only couple of lines of the log file is coming in the
sink, but in the source side has more lines.

I dont whether I put my question is correctly worder.

My configruation files below.

Apache configuration
-----------------------------

CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
/mnt/je/logs/apache/jesites/epicenter/access/%Y-%m-%d-%H-%M-%S-access.log
5M" cookie env=!dontlog
CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l
/mnt/je/logs/apache/jesites/epicenter/access/%Y-%m-%d-%H-%M-%S-access_assets.log
5M" combined env=dontlog
ErrorLog "|/opt/bitnami/apache2/bin/rotatelogs -l
/mnt/je/logs/apache/jesites/epicenter/error/%Y-%m-%d-%H-%M-%S-errorlog.log
5M"
Flume configuration
---------------------------
#source is of type spooling directory - epicenter access logs
agent1.sources.spooldir-epi-access.channels = ch1
agent1.sources.spooldir-epi-access.type = spooldir
agent1.sources.spooldir-epi-access.spoolDir /mnt/je/logs/apache/jesites/epicenter/access
agent1.sources.spooldir-epi-access.interceptors = i1 hostname type
agent1.sources.spooldir-epi-access.interceptors.i1.type = timestamp
agent1.sources.spooldir-epi-access.interceptors.hostname.type = host
agent1.sources.spooldir-epi-access.interceptors.hostname.useIP = false
agent1.sources.spooldir-epi-access.interceptors.hostname.preserveExisting true
agent1.sources.spooldir-epi-access.interceptors.type.type = static
agent1.sources.spooldir-epi-access.interceptors.type.key = type
agent1.sources.spooldir-epi-access.interceptors.type.value = epi-access
agent1.sources.spooldir-epi-access.fileHeader = true
--
Regards,

Robert George

http://justEat.in | [EMAIL PROTECTED] | +919986442677