Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Problem Events


Copy link to this message
-
Problem Events
Jeremy Karlson 2013-07-24, 21:52
Hi everyone.  My Flume adventures continue.

I'm in a situation now where I have a channel that's filling because a
stubborn message is stuck.  The sink won't accept it (for whatever reason;
I can go into detail but that's not my point here).  This just blocks up
the channel entirely, because it goes back into the channel when the sink
refuses.  Obviously, this isn't ideal.

I'm wondering what mechanisms, if any, Flume has to deal with these
situations.  Things that come to mind might be:

1. Ditch the event after n attempts.
2. After n attempts, send the event to a "problem area" (maybe a different
source / sink / channel?)  that someone can look at later.
3. Some sort of mechanism that allows operators to manually kill these
messages.

I'm open to suggestions on alternatives as well.

Thanks.

-- Jeremy