Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: Calling C inside MR


Copy link to this message
-
Re: Calling C inside MR
Brock Noland 2012-12-03, 15:56
Hi,

Here is an example of how to call native methods from Java in a MR context:

https://github.com/brockn/hadoop-thumbnail

The most important item IMHO is that you have a clear separation of
concerns. Meaning that you can test the C code without java and test
the C+Java without MapReduce.

Brock

On Mon, Dec 3, 2012 at 7:52 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello Simone,
>
>      Thank you so much for the pointers. I am actually looking for some way
> using which this can be achieved without Streaming or Pipes (If possible at
> all).
>
> And I did not get why did you point out to the API page. Please let me know
> if you know about something that I could relate. Apologies for my ignorance.
>
> Many thanks.
>
> Regards,
>     Mohammad Tariq
>
>
>
> On Mon, Dec 3, 2012 at 7:11 PM, Simone Leo <[EMAIL PROTECTED]> wrote:
>>
>> You can use Hadoop Pipes:
>>
>> http://hadoop.apache.org/docs/stable/api/index.html
>> http://wiki.apache.org/hadoop/C%2B%2BWordCount
>>
>> Simone
>>
>>
>> On 12/03/2012 01:05 PM, Mohammad Tariq wrote:
>>>
>>> Hello list,
>>>
>>>            I have a tool (written in C) that performs some different
>>> types of operations and can be used as a command line utility. I had to
>>> write a similar tool, as we have moved towards Hadoop platform for most
>>> of the things.
>>>
>>> Till now I have taken this tool as reference  and written MR jobs
>>> corresponding to some the modules of this tool and they are working
>>> fine. But I am wasting a lot of time in this. So, I just wanted to ask
>>> if it is possible to call this tool through a MR job?? Somewhat like JNI
>>> kinda thing. (I hope it is, otherwise I have to write rest of things
>>> from scratch and we are running out of time).
>>>
>>> Many thanks.
>>>
>>> Regards,
>>>      Mohammad Tariq
>>>
>>
>> --
>> Simone Leo
>> Data Fusion - Distributed Computing
>> CRS4
>> POLARIS - Building #1
>> Piscina Manna
>> I-09010 Pula (CA) - Italy
>> e-mail: [EMAIL PROTECTED]
>> http://www.crs4.it
>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/