Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: Calling C inside MR


Copy link to this message
-
Re: Calling C inside MR
Hi,

Here is an example of how to call native methods from Java in a MR context:

https://github.com/brockn/hadoop-thumbnail

The most important item IMHO is that you have a clear separation of
concerns. Meaning that you can test the C code without java and test
the C+Java without MapReduce.

Brock

On Mon, Dec 3, 2012 at 7:52 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Hello Simone,
>
>      Thank you so much for the pointers. I am actually looking for some way
> using which this can be achieved without Streaming or Pipes (If possible at
> all).
>
> And I did not get why did you point out to the API page. Please let me know
> if you know about something that I could relate. Apologies for my ignorance.
>
> Many thanks.
>
> Regards,
>     Mohammad Tariq
>
>
>
> On Mon, Dec 3, 2012 at 7:11 PM, Simone Leo <[EMAIL PROTECTED]> wrote:
>>
>> You can use Hadoop Pipes:
>>
>> http://hadoop.apache.org/docs/stable/api/index.html
>> http://wiki.apache.org/hadoop/C%2B%2BWordCount
>>
>> Simone
>>
>>
>> On 12/03/2012 01:05 PM, Mohammad Tariq wrote:
>>>
>>> Hello list,
>>>
>>>            I have a tool (written in C) that performs some different
>>> types of operations and can be used as a command line utility. I had to
>>> write a similar tool, as we have moved towards Hadoop platform for most
>>> of the things.
>>>
>>> Till now I have taken this tool as reference  and written MR jobs
>>> corresponding to some the modules of this tool and they are working
>>> fine. But I am wasting a lot of time in this. So, I just wanted to ask
>>> if it is possible to call this tool through a MR job?? Somewhat like JNI
>>> kinda thing. (I hope it is, otherwise I have to write rest of things
>>> from scratch and we are running out of time).
>>>
>>> Many thanks.
>>>
>>> Regards,
>>>      Mohammad Tariq
>>>
>>
>> --
>> Simone Leo
>> Data Fusion - Distributed Computing
>> CRS4
>> POLARIS - Building #1
>> Piscina Manna
>> I-09010 Pula (CA) - Italy
>> e-mail: [EMAIL PROTECTED]
>> http://www.crs4.it
>
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB