Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Need for UDP / Multicast Source

Copy link to this message
Re: Need for UDP / Multicast Source
> Also it may be useful to turn up jmx monitoring and watch the channel counters using console.
Yeah, I've done this too.  Everything source->sink related seems to keep up.

> Can you see if something like 'netstat -su' on sources and destination flume nodes shows any problems.

Sorry, I should have done this earlier.  'packet receive errors' steadily increases on my Flume node while flume is running.


netstat -anu | grep 8420
udp6  228096      0        :::*

RecvQ is pretty full there!  

It looks to me like Flume isn't reading off of the UDP queue and do the channel put fast enough.  

> I found that because UDP can so easily drop data, I needed a thread
> dedicated to reading the events and then immediately hand them off
> another thread to do anything interesting. It's possible you are in
> this scenario.

Yeah, sigh.  Maybe so.  One of the reasons we are trying Flume is to get ourselves out of this situation though!