Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> how to pass a hdfs file to a c++ process


Copy link to this message
-
Re: how to pass a hdfs file to a c++ process
That is a normal use case.

I'd encourage you to use Java MR (even pig/hive).

If you really want to use your legacy app use streaming with a map cmd
such as 'hadoop fs -cat <file> | mylegacyexe'

Arun

Sent from my iPhone

On Aug 23, 2011, at 8:00 AM, Zhixuan Zhu <[EMAIL PROTECTED]> wrote:

> I'll actually invoke one executable from each of my map. Because this
> C++ program has been implemented and used in the past, I just want to
> integrate it to our Hadoop map/reduce without having to re-implement the
> process in java. So my map is going to be very simple with just calling
> the process and pass the input files.
>
> Thanks,
> Grace
>
> -----Original Message-----
> From: Arun C Murthy [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, August 23, 2011 9:51 AM
> To: [EMAIL PROTECTED]
> Subject: Re: how to pass a hdfs file to a c++ process
>
>
> On Aug 22, 2011, at 12:57 PM, Zhixuan Zhu wrote:
>
>> Hi All,
>>
>> I'm using hadoop-0.20.2 to try out some simple tasks. I asked a
> question
>> about FileInputFormat a few days ago and get some prompt replys from
>> this forum and it helped a lot. Thanks again! Now I have another
>> question. I'm trying to invoke a C++ process from my mapper for each
>> hdfs file in the input directory to achieve some parallel processing.
>
> That seems weird - why aren't you using more maps and one file per-map?
>
>> But how do I pass the file to the program? I would want to do
> something
>> like the following in my mapper:
>
> IAC, libhdfs is one way to do HDFS ops via c/c++.
>
> Arun
>
>>
>> Process lChldProc = Runtime.getRuntime().exec("myprocess -file
>> $filepath");
>>
>> How do I pass the hdfs filesystem to an outside process like that? Is
>> HadoopStreaming the direction I should go?
>>
>> Thanks very much for any reply in advance.
>>
>> Best,
>> Grace
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB