Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Equivalent of Decorators in Flume NG


Copy link to this message
-
Re: Equivalent of Decorators in Flume NG
Hi Harish,
Why do you want to do that? Can you describe your use case?

Regards,
Mike

On Tue, Oct 2, 2012 at 1:28 PM, Harish Mandala <[EMAIL PROTECTED]>wrote:

> Hello,
>
> Alright, so maybe interceptors were not exactly what I wanted.
>
> It seems the number of events going into an interceptor must equal the
> number coming out. However, what if I need to take out the data from a
> certain event, and create multiple events from subsets of the data which
> would then be multiplexed using the selector to different locations. Would
> the job of splitting one event into many best be done in a Source or Sink?
>
> I was contemplating modifying the AvroSource or AvroSink for my purposes.
> However, it seems the sink also tallies output event counts and input event
> counts, and makes sure they're the same. That leaves me the option of
> writing a custom source based off the AvroSource. Is my thinking correct?
>
> Thanks,
> Harish
>
> On Mon, Oct 1, 2012 at 6:45 PM, Harish Mandala <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Percy,
> >
> > Thanks! Interceptors seem good enough.
> >
> > Regards,
> > Harish
> >
> >
> > On Mon, Oct 1, 2012 at 6:32 PM, Mike Percy <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Harish,
> >> At this time Flume NG doesn't support unbatching or sink-side plugins.
> >> Interceptors provide source-side tagging, filtering, and transformation
> >> capability, however.
> >>
> >> Regards,
> >> Mike
> >>
> >>
> >> On Mon, Oct 1, 2012 at 3:23 PM, Harish Mandala <[EMAIL PROTECTED]
> >> >wrote:
> >>
> >> > Hello,
> >> >
> >> > Am I right in thinking Flume NG no longer has the concept of Sink
> >> > Decorators? I wanted to do some custom deserialization on incoming
> event
> >> > data, and split one event into several (De-batching and re-routing).
> >> What's
> >> > the best way to implement this in Flume NG?
> >> >
> >> > Thanks,
> >> > Harish
> >> >
> >>
> >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB