Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Dupes


Copy link to this message
-
Re: Dupes
Cochran, David 2013-04-05, 14:22
Hi Israel,

I copied out the portions of my config that pertain to the server that I'm
seeing this bad behavior from (and sanitized it a little)  Otherwise my
config is like 1400 lines now, trying to stay with a single config and dist
it out to each server for consistency to save headaches.

Dave

# define sources, channels, and sinks for each log file
node_usvsm01.sources = source21061 source21062 source21063
node_usvsm01.channels = channel21061 channel21062 channel21063
node_usvsm01.sinks = sink21061 sink21062 sink21063

# source file usvsm01 - smaccess.log
node_usvsm01.sources.source21061.type = exec
node_usvsm01.sources.source21061.command = tail -F
/opt/siteminder/CA/log/smaccess.log
node_usvsm01.sources.source21061.channels = channel21061
# source file imsnolusvsm01 - smps.log
node_usvsm01.sources.source21062.type = exec
node_usvsm01.sources.source21062.command = tail -F
/opt/siteminder/CA/log/smps.log
node_usvsm01.sources.source21062.channels = channel21062
# source file imsnolusvsm01 - smtracedefault.log
node_usvsm01.sources.source21063.type = exec
node_usvsm01.sources.source21063.command = tail -F
/opt/siteminder/CA/log/smtracedefault.log
node_usvsm01.sources.source21063.channels = channel21063

node_usvsm01.channels.channel21061.type = memory
node_usvsm01.channels.channel21061.capacity = 100000
node_usvsm01.channels.channel21061.transactionCapactiy = 1000
node_usvsm01.channels.channel21062.type = memory
node_usvsm01.channels.channel21062.capacity = 100000
node_usvsm01.channels.channel21062.transactionCapactiy = 1000
node_usvsm01.channels.channel21063.type = memory
node_usvsm01.channels.channel21063.capacity = 100000
node_usvsm01.channels.channel21063.transactionCapactiy = 1000

# send channels --> flume @ usvinf01
node_usvsm01.sinks.sink21061.type = avro
node_usvsm01.sinks.sink21061.channel = channel21061
node_usvsm01.sinks.sink21061.hostname = usinf01
node_usvsm01.sinks.sink21061.port = 21061
node_usvsm01.sinks.sink21062.type = avro
node_usvsm01.sinks.sink21062.channel = channel21062
node_usvsm01.sinks.sink21062.hostname = usinf01
node_usvsm01.sinks.sink21062.port = 21062
node_usvsm01.sinks.sink21063.type = avro
node_usvsm01.sinks.sink21063.channel = channel21063
node_usvsm01.sinks.sink21063.hostname = usinf01
node_usvsm01.sinks.sink21063.port = 21063
node102.sources = source21061 source21062 source21063
node102.channels = channel21061 channel21062 channel21063
node102.sinks = sink21061 sink21062 sink21063

#  - usvsm01 -
# source file usvsm01 - smaccess.log
node102.sources.source21061.type = avro
node102.sources.source21061.bind = 0.0.0.0
node102.sources.source21061.port = 21061
node102.sources.source21061.channels = channel21061
# source file usvsm01 - smps.log
node102.sources.source21062.type = avro
node102.sources.source21062.bind = 0.0.0.0
node102.sources.source21062.port = 21062
node102.sources.source21062.channels = channel21062
# source file usvsm01 - smtracedefault.log
node102.sources.source21063.type = avro
node102.sources.source21063.bind = 0.0.0.0
node102.sources.source21063.port = 21063
node102.sources.source21063.channels = channel21063

#  - usvsm01 -
node102.channels.channel21061.type = memory
node102.channels.channel21061.capacity = 100000
node102.channels.channel21061.transactionCapactiy = 1000
node102.channels.channel21062.type = memory
node102.channels.channel21062.capacity = 100000
node102.channels.channel21062.transactionCapactiy = 1000
node102.channels.channel21063.type = memory
node102.channels.channel21063.capacity = 100000
node102.channels.channel21063.transactionCapactiy = 1000

# usvsm01 -
# source file usvsm01 - smaccess.log
node102.sinks.sink21061.type = FILE_ROLL
node102.sinks.sink21061.channel = channel21061
node102.sinks.sink21061.sink.directory /flume_logs/usvsm01/siteminder/smaccess_log
node102.sinks.sink21061.sink.rollInterval = 86400
node102.sinks.sink21061.sink.serializer = TEXT
# source file usvsm01 - smps.log
node102.sinks.sink21062.type = FILE_ROLL
node102.sinks.sink21062.channel = channel21062
node102.sinks.sink21062.sink.directory /flume_logs/usvsm01/siteminder/smps_log
node102.sinks.sink21062.sink.rollInterval = 86400
node102.sinks.sink21062.sink.serializer = TEXT
# source file usvsm01 - smtracedefault.log
node102.sinks.sink21063.type = FILE_ROLL
node102.sinks.sink21063.channel = channel21063
node102.sinks.sink21063.sink.directory /flume_logs/usvsm01/siteminder/smtracedefault_log
node102.sinks.sink21063.sink.rollInterval = 86400
node102.sinks.sink21063.sink.serializer = TEXT

On Fri, Apr 5, 2013 at 9:00 AM, Israel Ekpo <[EMAIL PROTECTED]> wrote: