Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Avro sink to source is too slow


Copy link to this message
-
Re: Avro sink to source is too slow
My thoughts...You have 4 sinks draining the same channel and each has a
batch size 1000. Since they will contend on the same channel & *assuming*
events are evenly distributed among the sinks, there is potential for some
starvation happening in the sinks as their batch sizes may not be reached
until about 4 batches  are inserted by the source. I dont know if there is
a good thumb rule here.

try these:
-  See if sink batch size of 250 helps.
-  Using a single avro sink instead of 4 with batch size of 1k.
-  Replacing the  avro sink with the null sink on the first agent and take
a measurement. it would be good to ensure spool source is not the bottle
neck.

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB