Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> CPU rise when channel is full

Copy link to this message
CPU rise when channel is full

I'm using a classic Flume topology where I have many app servers which I
want to collect logs from.
They all send the data through an AvroSink=>Avro source to a collector

In each app server I use: Spool dir=>File channel=>Avro Sink

My problem is that I want to avoid any major influence on app servers while
using Flume.
Most of the time this is true, however when the filechannel used in the app
server is filled for some reason, the CPU used by the Flume java process is
rising and gets to 100% CPU, then drops down and then up again.

I don't want to interrupt the app server work because of a problem in the
logging flow.
Is there any way to avoid this behavior? Why does Flume use so much CPU
when the file channel is full?