Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Streaming Hadoop using C


Copy link to this message
-
Re: Streaming Hadoop using C
Charles Earl 2012-02-29, 19:35
Mark,
So if I understand, it is more the memory management that you are interested in, rather than a need to run an existing C or C++ application in MapReduce platform?
Have you done profiling of the application?
C
On Feb 29, 2012, at 2:19 PM, Mark question wrote:

> Thanks Charles .. I'm running Hadoop for research to perform duplicate
> detection methods. To go deeper, I need to understand what's slowing my
> program, which usually starts with analyzing memory to predict best input
> size for map task. So you're saying piping can help me control memory even
> though it's running on VM eventually?
>
> Thanks,
> Mark
>
> On Wed, Feb 29, 2012 at 11:03 AM, Charles Earl <[EMAIL PROTECTED]>wrote:
>
>> Mark,
>> Both streaming and pipes allow this, perhaps more so pipes at the level of
>> the mapreduce task. Can you provide more details on the application?
>> On Feb 29, 2012, at 1:56 PM, Mark question wrote:
>>
>>> Hi guys, thought I should ask this before I use it ... will using C over
>>> Hadoop give me the usual C memory management? For example, malloc() ,
>>> sizeof() ? My guess is no since this all will eventually be turned into
>>> bytecode, but I need more control on memory which obviously is hard for
>> me
>>> to do with Java.
>>>
>>> Let me know of any advantages you know about streaming in C over hadoop.
>>> Thank you,
>>> Mark
>>
>>