Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> CPU rise when channel is full


Copy link to this message
-
CPU rise when channel is full
Hi,

I'm using a classic Flume topology where I have many app servers which I
want to collect logs from.
They all send the data through an AvroSink=>Avro source to a collector
machine.

In each app server I use: Spool dir=>File channel=>Avro Sink

My problem is that I want to avoid any major influence on app servers while
using Flume.
Most of the time this is true, however when the filechannel used in the app
server is filled for some reason, the CPU used by the Flume java process is
rising and gets to 100% CPU, then drops down and then up again.

I don't want to interrupt the app server work because of a problem in the
logging flow.
Is there any way to avoid this behavior? Why does Flume use so much CPU
when the file channel is full?

Thanks
Anat
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB