Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Allowing custom FlumeFormatter to be used when writing HDFS SequenceFiles


+
バーチャル クリストファー 2012-08-10, 09:22
Copy link to this message
-
Re: Allowing custom FlumeFormatter to be used when writing HDFS SequenceFiles
Hi Chris,
It would be great if you could file a JIRA to track this. JIRA is up again.

Regards,
Mike

On Fri, Aug 10, 2012 at 2:22 AM, バーチャル クリストファー
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> I have rewritten the HDFS event sink's HDFSFormatterFactory so that I
> can use my own FlumeFormatter implementation to write events to
> SequenceFiles. I did this because the standard HDFSWritableFormatter
> discards all headers apart from the timestamp, whereas I want to write
> all headers to HDFS.
>
> The code is available for review
> here:<https://github.com/cb372/flume/tree/custom-hdfs-formatter>
>
> https://github.com/cb372/flume/tree/custom-hdfs-formatter
>
> To use it, you just pass in the FlumeFormatter implementation's
> classname in the config, similar to the way you specify a custom
> EventSerializer.
>
> e.g.
> agent_foo.sinks.hdfs-sink.writeFormat=com.mycompany.flume.MyCustomFormatter
>
> The class must have a public zero-argument constructor.
>
> Please let me know what you think,
>
> Chris.
>
> PS: I would have filed a Jira (maybe there already is one?), but the
> Jira server is down at the moment.
>
+
バーチャル クリストファー 2012-08-16, 00:13