Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Dupes


Hi Israel,

I copied out the portions of my config that pertain to the server that I'm
seeing this bad behavior from (and sanitized it a little)  Otherwise my
config is like 1400 lines now, trying to stay with a single config and dist
it out to each server for consistency to save headaches.

Dave

# define sources, channels, and sinks for each log file
node_usvsm01.sources = source21061 source21062 source21063
node_usvsm01.channels = channel21061 channel21062 channel21063
node_usvsm01.sinks = sink21061 sink21062 sink21063

# source file usvsm01 - smaccess.log
node_usvsm01.sources.source21061.type = exec
node_usvsm01.sources.source21061.command = tail -F
/opt/siteminder/CA/log/smaccess.log
node_usvsm01.sources.source21061.channels = channel21061
# source file imsnolusvsm01 - smps.log
node_usvsm01.sources.source21062.type = exec
node_usvsm01.sources.source21062.command = tail -F
/opt/siteminder/CA/log/smps.log
node_usvsm01.sources.source21062.channels = channel21062
# source file imsnolusvsm01 - smtracedefault.log
node_usvsm01.sources.source21063.type = exec
node_usvsm01.sources.source21063.command = tail -F
/opt/siteminder/CA/log/smtracedefault.log
node_usvsm01.sources.source21063.channels = channel21063

node_usvsm01.channels.channel21061.type = memory
node_usvsm01.channels.channel21061.capacity = 100000
node_usvsm01.channels.channel21061.transactionCapactiy = 1000
node_usvsm01.channels.channel21062.type = memory
node_usvsm01.channels.channel21062.capacity = 100000
node_usvsm01.channels.channel21062.transactionCapactiy = 1000
node_usvsm01.channels.channel21063.type = memory
node_usvsm01.channels.channel21063.capacity = 100000
node_usvsm01.channels.channel21063.transactionCapactiy = 1000

# send channels --> flume @ usvinf01
node_usvsm01.sinks.sink21061.type = avro
node_usvsm01.sinks.sink21061.channel = channel21061
node_usvsm01.sinks.sink21061.hostname = usinf01
node_usvsm01.sinks.sink21061.port = 21061
node_usvsm01.sinks.sink21062.type = avro
node_usvsm01.sinks.sink21062.channel = channel21062
node_usvsm01.sinks.sink21062.hostname = usinf01
node_usvsm01.sinks.sink21062.port = 21062
node_usvsm01.sinks.sink21063.type = avro
node_usvsm01.sinks.sink21063.channel = channel21063
node_usvsm01.sinks.sink21063.hostname = usinf01
node_usvsm01.sinks.sink21063.port = 21063
node102.sources = source21061 source21062 source21063
node102.channels = channel21061 channel21062 channel21063
node102.sinks = sink21061 sink21062 sink21063

#  - usvsm01 -
# source file usvsm01 - smaccess.log
node102.sources.source21061.type = avro
node102.sources.source21061.bind = 0.0.0.0
node102.sources.source21061.port = 21061
node102.sources.source21061.channels = channel21061
# source file usvsm01 - smps.log
node102.sources.source21062.type = avro
node102.sources.source21062.bind = 0.0.0.0
node102.sources.source21062.port = 21062
node102.sources.source21062.channels = channel21062
# source file usvsm01 - smtracedefault.log
node102.sources.source21063.type = avro
node102.sources.source21063.bind = 0.0.0.0
node102.sources.source21063.port = 21063
node102.sources.source21063.channels = channel21063

#  - usvsm01 -
node102.channels.channel21061.type = memory
node102.channels.channel21061.capacity = 100000
node102.channels.channel21061.transactionCapactiy = 1000
node102.channels.channel21062.type = memory
node102.channels.channel21062.capacity = 100000
node102.channels.channel21062.transactionCapactiy = 1000
node102.channels.channel21063.type = memory
node102.channels.channel21063.capacity = 100000
node102.channels.channel21063.transactionCapactiy = 1000

# usvsm01 -
# source file usvsm01 - smaccess.log
node102.sinks.sink21061.type = FILE_ROLL
node102.sinks.sink21061.channel = channel21061
node102.sinks.sink21061.sink.directory /flume_logs/usvsm01/siteminder/smaccess_log
node102.sinks.sink21061.sink.rollInterval = 86400
node102.sinks.sink21061.sink.serializer = TEXT
# source file usvsm01 - smps.log
node102.sinks.sink21062.type = FILE_ROLL
node102.sinks.sink21062.channel = channel21062
node102.sinks.sink21062.sink.directory /flume_logs/usvsm01/siteminder/smps_log
node102.sinks.sink21062.sink.rollInterval = 86400
node102.sinks.sink21062.sink.serializer = TEXT
# source file usvsm01 - smtracedefault.log
node102.sinks.sink21063.type = FILE_ROLL
node102.sinks.sink21063.channel = channel21063
node102.sinks.sink21063.sink.directory /flume_logs/usvsm01/siteminder/smtracedefault_log
node102.sinks.sink21063.sink.rollInterval = 86400
node102.sinks.sink21063.sink.serializer = TEXT

On Fri, Apr 5, 2013 at 9:00 AM, Israel Ekpo <[EMAIL PROTECTED]> wrote:

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB