Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: Calling C inside MR

Copy link to this message
Re: Calling C inside MR

Here is an example of how to call native methods from Java in a MR context:


The most important item IMHO is that you have a clear separation of
concerns. Meaning that you can test the C code without java and test
the C+Java without MapReduce.


On Mon, Dec 3, 2012 at 7:52 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello Simone,
>      Thank you so much for the pointers. I am actually looking for some way
> using which this can be achieved without Streaming or Pipes (If possible at
> all).
> And I did not get why did you point out to the API page. Please let me know
> if you know about something that I could relate. Apologies for my ignorance.
> Many thanks.
> Regards,
>     Mohammad Tariq
> On Mon, Dec 3, 2012 at 7:11 PM, Simone Leo <[EMAIL PROTECTED]> wrote:
>> You can use Hadoop Pipes:
>> http://hadoop.apache.org/docs/stable/api/index.html
>> http://wiki.apache.org/hadoop/C%2B%2BWordCount
>> Simone
>> On 12/03/2012 01:05 PM, Mohammad Tariq wrote:
>>> Hello list,
>>>            I have a tool (written in C) that performs some different
>>> types of operations and can be used as a command line utility. I had to
>>> write a similar tool, as we have moved towards Hadoop platform for most
>>> of the things.
>>> Till now I have taken this tool as reference  and written MR jobs
>>> corresponding to some the modules of this tool and they are working
>>> fine. But I am wasting a lot of time in this. So, I just wanted to ask
>>> if it is possible to call this tool through a MR job?? Somewhat like JNI
>>> kinda thing. (I hope it is, otherwise I have to write rest of things
>>> from scratch and we are running out of time).
>>> Many thanks.
>>> Regards,
>>>      Mohammad Tariq
>> --
>> Simone Leo
>> Data Fusion - Distributed Computing
>> CRS4
>> POLARIS - Building #1
>> Piscina Manna
>> I-09010 Pula (CA) - Italy
>> e-mail: [EMAIL PROTECTED]
>> http://www.crs4.it

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/