Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - How to write full event include headers to hdfs sink


Copy link to this message
-
Re: How to write full event include headers to hdfs sink
Hari Shreedharan 2012-09-01, 20:34
Hi,  

Since users would want to write the data in a format their processing code would understand, Flume does not impose a format on the data it writes. Instead Flume allows you to plugin your own code to serialize each event. By default, the serializer used is BodyTextEventSerializer which simply stringifies the body and writes it out. Flume does have an Avro serializer too, which writes the headers and body in Avro format.  

If you need the data in any other format you would need to write a serializer that serializes the data.  

Thanks
Hari
--  
Hari Shreedharan
On Saturday, September 1, 2012 at 2:32 AM, Kevin Lee wrote:

> Hi all,
> When i use Flume-OG, hdfs sink will get full event like follow text.
> head -1 varnishncsa.log | python -mjson.tool { "body": "xxxxxxxxx", "fields": { "rolltag": "20120427-234205010-0400.418723114906325.00004966", "tailSrcFile": "varnishncsa.log" }, "host": "ip-10-170-147-62.compute.internal", "nanos": 5164871621338860, "pri": "INFO", "timestamp": 1335584717134 }  
> But when i using flume-NG. If i’m use a logger sink, it will give me full event, like bellow.
> collector config:
> agent.sinks.hdfs-write.type = logger  
> agent config:
> agent.sources.exec-agent.interceptors.hostname.type = org.apache.flume.interceptor.HostInterceptor$Builder agent.sources.exec-agent.interceptors.hostname.preserveExisting = true agent.sources.exec-agent.interceptors.hostname.useIP = false  
> >  
> > 12/09/01 07:51:41 INFO sink.LoggerSink: Event: { headers:{timestamp=1346485898919, host=ip-10-34-4-55.ec2.internal} body: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 aaaaaaaaaaaaaaaa }
>  
> But when i use hdfs sink, i can’t get the full event, thanks in advance for any help.
> Thanks,  
> - Kevin