Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> "single source - multi channel" scenario  and applying interceptor while writing to only one channel and not on others...possible approaches


+
Jagadish Bihani 2013-04-16, 06:36
+
Jagadish Bihani 2013-04-17, 06:12
+
Jeff Lord 2013-04-19, 16:34
Copy link to this message
-
Re: "single source - multi channel" scenario and applying interceptor while writing to only one channel and not on others...possible approaches
Jagadish,

Here is an example of how to write a custom serializer.

https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MyCustomSerializer.java

-Jeff
On Fri, Apr 19, 2013 at 9:34 AM, Jeff Lord <[EMAIL PROTECTED]> wrote:

> Hi Jagadish,
>
> Have you considered using a custom event serializer to modify your event?
> Its possible to replicate your flow using two channels and then have one
> sink that implements a custom serializer to modify the event.
>
> -Jeff
>
>
> On Tue, Apr 16, 2013 at 11:12 PM, Jagadish Bihani <
> [EMAIL PROTECTED]> wrote:
>
>> Hi
>>
>> If anybody has any inputs on this that will surely help.
>>
>> Regards,
>> Jagadish
>>
>>
>> On 04/16/2013 12:06 PM, Jagadish Bihani wrote:
>>
>>> Hi
>>>
>>> We have a use case in which
>>> 1. spooling source reads data.
>>> 2. It needs to write events into multiple channels. It should apply
>>> interceptor only when putting into one channel and should put
>>> the event as it is while putting into another channel.
>>>
>>> Possible approach we have thought:
>>>
>>> 1. Create  2 different sources and then apply interceptor on one and dont
>>> apply on other. But that duplicates reads and increases IO.
>>>
>>> Is there any better way of achieving this use case?
>>>
>>> Regards,
>>> Jagadish
>>>
>>>
>>
>
+
Connor Woodson 2013-04-23, 06:52
+
Connor Woodson 2013-04-23, 06:57
+
Jagadish Bihani 2013-04-23, 09:02
+
Israel Ekpo 2013-04-23, 14:15
+
Connor Woodson 2013-04-26, 05:15
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB