Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Problem Events


Copy link to this message
-
Re: Problem Events
Roshan Naik 2013-07-24, 22:36
if you have a way to identify such events.. you may be able to use the
Regex interceptor to toss them out before they get into the channel.
On Wed, Jul 24, 2013 at 2:52 PM, Jeremy Karlson <[EMAIL PROTECTED]>wrote:

> Hi everyone.  My Flume adventures continue.
>
> I'm in a situation now where I have a channel that's filling because a
> stubborn message is stuck.  The sink won't accept it (for whatever reason;
> I can go into detail but that's not my point here).  This just blocks up
> the channel entirely, because it goes back into the channel when the sink
> refuses.  Obviously, this isn't ideal.
>
> I'm wondering what mechanisms, if any, Flume has to deal with these
> situations.  Things that come to mind might be:
>
> 1. Ditch the event after n attempts.
> 2. After n attempts, send the event to a "problem area" (maybe a different
> source / sink / channel?)  that someone can look at later.
> 3. Some sort of mechanism that allows operators to manually kill these
> messages.
>
> I'm open to suggestions on alternatives as well.
>
> Thanks.
>
> -- Jeremy
>