Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - custome serializer


+
Chhaya Vishwakarma 2013-01-11, 09:23
+
shekhar sharma 2013-01-11, 14:07
+
Connor Woodson 2013-01-11, 19:37
+
Chhaya Vishwakarma 2013-01-14, 04:56
Copy link to this message
-
Re: custome serializer
Connor Woodson 2013-01-14, 05:40
For what specifically? I provided instructions for how to use the
HeaderAndText serializer; is that what you want? If you have a specific
question, I'll be happy to answer it.

For how to create a custom serializer, I'll try to add some more details.
This assumes you know a little Java:

Do the git clone to get the latest Flume: git clone
http://github.com/apache/flume
Go into the Flume directory (I think it's just called flume); there will be
a bunch of flume-ng-<stuff> and a bin folder and so forth.
>From here, copy the file:
flume-ng-core/src/main/java/org/apache/flume/serialization/HeaderAndBodyTextEventSerializer.javainto
the file:
flume-ng-core/src/main/java/org/apache/flume/serialization/CustomHeaderAndBodyTextEventSerializer.java
Change the class name/constructor of that new file to
CustomHeaderAndBodyTextEventSerializer
Change the Builder inner-class appropriately (and get rid of any errors
this all might have caused)
Modify it how ever you want (if there's some specific modification you want
to do, just ask and I can try and help out)
Save, of course
Now you need to compile/package Flume: export JAVA_HOME=<your java
home> ; export
MAVEN_OPTS="-Xmx512M -XX:MaxPermSize=512M" ; mvn package -DskipTests
Take the package from the 'target' directory called (I think)
apache-flume-1.4.0-SNAPSHOT-bin.tar.gz and move it somewhere
Go to wherever you moved that file
Decompress it: tar xzf apache-flume-1.4.0-SNAPSHOT-bin.tar.gz
Delete the .tar.gz if you want
In your Flume config file, change the serializer property of your sink to
be:
...serializer org.apache.flume.serialization.CustomHeaderAndBodyTextEventSerializer$Builder
And run flume.

I think that's a comprehensive guide to modifying Flume's code; let me know
of any specific questions. If the 'mvn package' doesn't work it's probably
a result of compilation/syntax errors in your new class. If Flume doesn't
run properly, it would likely be due to a bad config file or runtime issues
in your new class.

- Connor

On Sun, Jan 13, 2013 at 8:56 PM, Chhaya Vishwakarma <
[EMAIL PROTECTED]> wrote:

> Can you share any example? ****
>
> ** **
>
> *From:* shekhar sharma [mailto:[EMAIL PROTECTED]]
> *Sent:* Friday, January 11, 2013 7:37 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: custome serializer****
>
> ** **
>
> I think you can implement your custome sink, where you can take event body
> and header if any to HDFS...****
>
>  ****
>
> Regardsm****
>
> Som****
>
> On Fri, Jan 11, 2013 at 2:53 PM, Chhaya Vishwakarma <
> [EMAIL PROTECTED]> wrote:****
>
> Hi,****
>
>  ****
>
> How can I write custom serializer to write event body and header to HDFS
> now I am getting only log messages which are written on HDFS. Timestamp and
> other information is not coming.****
>
>  ****
>
>  ****
>
> Regards,****
>
> Chhaya Vishwakarma****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> ** **
> ------------------------------
>
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"****
>
> ** **
>