Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to process different types of avro schema


Copy link to this message
-
How to process different types of avro schema
Hi All,****

In my application I am getting a stream of avro events. This stream
contains different types of avro events belonging to different schemas. I
was wondering what is the right way to process this data and do analytics
on top of this. Can I use hive? I did study the avro serde that could be
used to decode avro data and I’m thinking I need to transform the input
stream into (multiple) entries belonging to different tables. For this I’m
considering using a mapper job that would extract these events type by type
and then we could use hive on top of these separate schemas. I’m wondering
if anyone has dealt with such scenario before and if this approach would
work with decent performance?****

Alternative way is to use all the logic in M-R code for the analytics that
we want to do on top of this data. Please advise.****

Thanks in advance.****

Sourabh