Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - "single source - multi channel" scenario  and applying interceptor while writing to only one channel and not on others...possible approaches


Copy link to this message
-
Re: "single source - multi channel" scenario and applying interceptor while writing to only one channel and not on others...possible approaches
Jeff Lord 2013-04-19, 18:14
Jagadish,

Here is an example of how to write a custom serializer.

https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MyCustomSerializer.java

-Jeff
On Fri, Apr 19, 2013 at 9:34 AM, Jeff Lord <[EMAIL PROTECTED]> wrote:

> Hi Jagadish,
>
> Have you considered using a custom event serializer to modify your event?
> Its possible to replicate your flow using two channels and then have one
> sink that implements a custom serializer to modify the event.
>
> -Jeff
>
>
> On Tue, Apr 16, 2013 at 11:12 PM, Jagadish Bihani <
> [EMAIL PROTECTED]> wrote:
>
>> Hi
>>
>> If anybody has any inputs on this that will surely help.
>>
>> Regards,
>> Jagadish
>>
>>
>> On 04/16/2013 12:06 PM, Jagadish Bihani wrote:
>>
>>> Hi
>>>
>>> We have a use case in which
>>> 1. spooling source reads data.
>>> 2. It needs to write events into multiple channels. It should apply
>>> interceptor only when putting into one channel and should put
>>> the event as it is while putting into another channel.
>>>
>>> Possible approach we have thought:
>>>
>>> 1. Create  2 different sources and then apply interceptor on one and dont
>>> apply on other. But that duplicates reads and increases IO.
>>>
>>> Is there any better way of achieving this use case?
>>>
>>> Regards,
>>> Jagadish
>>>
>>>
>>
>