Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Converting text to avro in Flume


Copy link to this message
-
Re: Converting text to avro in Flume
Deepak Subhramanian 2013-10-04, 13:52
I tried using the HDFS Sink to generate the avro file by using the
serializer as avro_event. But it is not generating avro file. But a
sequence file. Is it not suppose to generate a avro file with default
schema ?  Or do I have to generate the avro data from text in my
HTTPHandler source ?

 "{ \"type\":\"record\", \"name\": \"Event\", \"fields\": [" +

      " {\"name\": \"headers\", \"type\": { \"type\": \"map\", \"values\":
\"string\" } }, " +
      " {\"name\": \"body\", \"type\": \"bytes\" } ] }");
On Thu, Oct 3, 2013 at 3:36 PM, Deepak Subhramanian <
[EMAIL PROTECTED]> wrote:

> Hi ,
>
> I want to convert xml files in text to an avro file and store it in hdfs .
> I get the xml files as a post request. I extended the  HTTPHandler to
> process the XML post request. Do I have to convert the data in text to avro
> in HTTPHandler or does the Avro Sink or HDFSSink convert it directly to
> avro with some configuration details. I want to store the entire xml string
> in an avro variable.
>
> Thanks in advance for any inputs.
> Deepak Subhramanian
>

--
Deepak Subhramanian