Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Lock contention in FileChannel


Copy link to this message
-
Re: Lock contention in FileChannel
What is sending the events to the avro source?
On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <[EMAIL PROTECTED]> wrote:

> Here's the config:
> # define channels, one for each disk
>
>
>
>
> agent1.channels.ch1.type = FILE
> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
> agent1.channels.ch1.dataDirs = /flume1/data
> agent1.channels.ch1.maxFileSize = 375809638400
> agent1.channels.ch1.capacity = 75000000
> agent1.channels.ch1.transactionCapacity = 4000
>
> agent1.channels.ch2.type = FILE
> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
> agent1.channels.ch2.dataDirs = /flume2/data
> agent1.channels.ch2.maxFileSize = 375809638400
> agent1.channels.ch2.capacity = 75000000
> agent1.channels.ch2.transactionCapacity = 4000
>
>
>
> # Define an Avro source named avroSource1
> # Each sink can connect to only one channel.
> # Connect it to channel ch1. Load balance it to 2 avroSinks
>
>
> agent1.sources.avroSource1.channels = ch1
> agent1.sources.avroSource1.type = avro
> agent1.sources.avroSource1.bind = 0.0.0.0
> agent1.sources.avroSource1.port = <port>
>
>
>
>
> agent1.sinks.avroSink1-1-1.type = avro
> agent1.sinks.avroSink1-1-1.channel = ch1
> agent1.sinks.avroSink1-1-1.hostname = <hostname>
> agent1.sinks.avroSink1-1-1.port = <port>
> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
> agent1.sinks.avroSink1-1-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink1-2-1.type = avro
> agent1.sinks.avroSink1-2-1.channel = ch1
> agent1.sinks.avroSink1-2-1.hostname = <hostname>
> agent1.sinks.avroSink1-2-1.port = <port>
> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
> agent1.sinks.avroSink1-2-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink1-3-1.type = avro
> agent1.sinks.avroSink1-3-1.channel = ch1
> agent1.sinks.avroSink1-3-1.hostname = <hostname>
> agent1.sinks.avroSink1-3-1.port = <port>
> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
> agent1.sinks.avroSink1-3-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink1-4-1.type = avro
> agent1.sinks.avroSink1-4-1.channel = ch1
> agent1.sinks.avroSink1-4-1.hostname = <hostname>
> agent1.sinks.avroSink1-4-1.port = <port>
> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
> agent1.sinks.avroSink1-4-1.batch-size = 4000
>
>
>
> #Add the sink groups; load-balance between each group of sinks which round
> robin between different hops
> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
> avroSink1-4-1
> agent1.sinkgroups.group1.processor.type = load_balance
> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
> agent1.sinkgroups.group1.processor.backoff = true
>
>
> #End of set
>
> # Define an Avro source named avroSource2
> # Each sink can connect to only one channel.
> # Connect it to channel ch2. Load balance it to 2 avroSinks
>
>
> agent1.sources.avroSource2.channels = ch2
> agent1.sources.avroSource2.type = avro
> agent1.sources.avroSource2.bind = 0.0.0.0
> agent1.sources.avroSource2.port = <port>
>
>
>
>
> agent1.sinks.avroSink2-1-1.type = avro
> agent1.sinks.avroSink2-1-1.channel = ch2
> agent1.sinks.avroSink2-1-1.hostname = <hostname>
> agent1.sinks.avroSink2-1-1.port = <port>
> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
> agent1.sinks.avroSink2-1-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink2-2-1.type = avro
> agent1.sinks.avroSink2-2-1.channel = ch2
> agent1.sinks.avroSink2-2-1.hostname = <hostname>
> agent1.sinks.avroSink2-2-1.port = <port>
> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
> agent1.sinks.avroSink2-2-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink2-3-1.type = avro
> agent1.sinks.avroSink2-3-1.channel = ch2
> agent1.sinks.avroSink2-3-1.hostname = <hostname>
> agent1.sinks.avroSink2-3-1.port = <port>
> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
> agent1.sinks.avroSink2-3-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink2-4-1.type = avro
> agent1.sinks.avroSink2-4-1.channel = ch2
> agent1.sinks.avroSink2-4-1.hostname = <hostname>
> agent1.sinks.avroSink2-4-1.port = <port>
> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB