Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Is there any way to run C++ code with native avro libraries on Hadoop / YARN?


Copy link to this message
-
Is there any way to run C++ code with native avro libraries on Hadoop / YARN?
Hi everybody,

in our project we have a lot of C++ code which we'd like to run on Hadoop.
Because of the complexity of input and output structures for this code I'd
like to use AVRO as serialization/deserialization format. I figured out a
way to do that with custom Java input and output format classes, but this
way I deal with a lot of JSON on C++ side which leads to code complexity
and not optimal performance.

What I'd like to do is write a wrapper code which uses Pipes and native C++
AVRO libraries to read avro files. I understand how to write/read avro FILE
in C++, but in Pipes we don't have file we have a stream.

Is there any possible way to do aht I want or is it completely impossible
at the moment.

Here is a link with the same question on stack overflow I asked aerlier:
https://stackoverflow.com/questions/19399601/emitting-avro-format-from-pipes-in-hadoop

Thanks in advance,
Viacheslav
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB