Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Why does a Flume source need to recognize the format of the message?


+
Praveen Sripati 2013-10-22, 17:59
Copy link to this message
-
Re: Why does a Flume source need to recognize the format of the message?
Jarek Jarcec Cecho 2013-10-22, 18:07
Hi Praveen,
I think that there is a confusion between message and payload. Whereas Flume do not need to understand the payload structure, it do need to understand the message to understand what events (what payloads) are there with what headers. To put it differently, Flume do not need to understand structure of the data that you are sending (payload is just a byte array for Flume), but that unknown structure needs to be transferred via known protocol (such as AVRO RPC).

Jarcec

On Tue, Oct 22, 2013 at 06:59:17PM +0100, Praveen Sripati wrote:
> According to the Flume documentation
>
> >>    A Flume source consumes events delivered to it by an external source
> like a web server. The external source sends events to Flume in a format
> that is recognized by the target Flume source. For example, an Avro Flume
> source can be used to receive Avro events from Avro clients or other Flume
> agents in the flow that send events from an Avro sink.
>
> Why does a Flume source need to recognize or understand the format of the
> message? While all it does it does is to forward the message to one of the
> channel.
>
> Thanks,
> Praveen
+
Roshan Naik 2013-10-22, 20:51
+
Roshan Naik 2013-10-22, 21:21
+
dwight.marzolf@... 2013-10-22, 21:33
+
Roshan Naik 2013-10-22, 22:30
+
dwight.marzolf@... 2013-10-23, 14:48
+
Roshan Naik 2013-10-23, 18:51
+
dwight.marzolf@... 2013-10-23, 20:43
+
Paul Chavez 2013-10-23, 21:28
+
dwight.marzolf@... 2013-10-25, 13:23