Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - In which configuration file to configure the "fs.inmemory.size.mb" parameter?


Copy link to this message
-
Re: In which configuration file to configure the "fs.inmemory.size.mb" parameter?
Yu Li 2010-07-01, 09:33
Hi Sriguru,

Thanks for your comments. Do you know how to set this parameter?

Best Regards,
Carp

2010/7/1 Srigurunath Chakravarthi <[EMAIL PROTECTED]>:
> Carp,
>  IMHO, .20.x has it. fs.inmemory.size.mb is the reduce-side equivalent of io.sort.mb. In the reducer tasks, intermediate map output is collected into a buffer (who size is governed by this parameter's value), and data is flushed into files as (partially) sorted KVs.
>
>  These files will be re-merged if we end up with more than io.sort.factor number of files, else KVs will be served out of these files to the reduce function directly.
>
>  I don't know where in the code it is though, sorry.
>
> cheers,
> Sriguru
>
>
>>-----Original Message-----
>>From: Yu Li [mailto:[EMAIL PROTECTED]]
>>Sent: Thursday, July 01, 2010 1:12 PM
>>To: [EMAIL PROTECTED]
>>Subject: In which configuration file to configure the
>>"fs.inmemory.size.mb" parameter?
>>
>>Hi all,
>>
>>I looked through the "Cluster Setup" guide under link
>>http://hadoop.apache.org/common/docs/r0.20.1/cluster_setup.html and
>>found there's a "fs.inmemory.size.mb" parameter for specifying memory
>>allocated for the in-memory file-system used to merge map-outputs at
>>the reduces, and this parameter is set in the "core-site.xml". But
>>when I checked the "core-default.xml" under path
>>"$HADOOP_HOME/src/core/", I didn't find the parameter at all, nor
>>could I find the parameter through JTUI after lauching jobs.
>>Does anybody know about this parameter? Has it been removed from
>>release 0.20.X? If it hasn't been removed, how could I set the
>>parameter besides using the -D option? Thanks in advance.
>>
>>Best Regards,
>>Carp
>