Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Need for UDP / Multicast Source


Copy link to this message
-
Re: Need for UDP / Multicast Source
> Also it may be useful to turn up jmx monitoring and watch the channel counters using console.
Yeah, I've done this too.  Everything source->sink related seems to keep up.

> Can you see if something like 'netstat -su' on sources and destination flume nodes shows any problems.

Sorry, I should have done this earlier.  'packet receive errors' steadily increases on my Flume node while flume is running.

Also:

netstat -anu | grep 8420
udp6  228096      0 233.58.59.1:8420        :::*

RecvQ is pretty full there!  

It looks to me like Flume isn't reading off of the UDP queue and do the channel put fast enough.  

> I found that because UDP can so easily drop data, I needed a thread
> dedicated to reading the events and then immediately hand them off
> another thread to do anything interesting. It's possible you are in
> this scenario.

Yeah, sigh.  Maybe so.  One of the reasons we are trying Flume is to get ourselves out of this situation though!
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB