Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> In which configuration file to configure the "fs.inmemory.size.mb" parameter?


Copy link to this message
-
Re: In which configuration file to configure the "fs.inmemory.size.mb" parameter?
Hi Ted,

Thanks for your help. Another question is: if this parameter is not used in
the 0.20.X release and the "Cluster Setup" is not updated, is there any
parameter replacing this one? It's a useful parameter IMHO.

Anyone knows about this? Thanks in advance!

Best Regards,
Carp

2010/7/2 Ted Yu <[EMAIL PROTECTED]>

> I found https://issues.apache.org/jira/browse/HADOOP-6812
>
> You can add the following to core-site.xml:
> <property>
> <name>fs.inmemory.size.mb</name>
> <value>100</value>
> </property>
>
> Default value is 100:
>      int size = Integer.parseInt(conf.get("fs.inmemory.size.mb", "100"));
> ./src/core/org/apache/hadoop/fs/InMemoryFileSystem.java
>
> On Thu, Jul 1, 2010 at 2:33 AM, Yu Li <[EMAIL PROTECTED]> wrote:
>
> > Hi Sriguru,
> >
> > Thanks for your comments. Do you know how to set this parameter?
> >
> > Best Regards,
> > Carp
> >
> > 2010/7/1 Srigurunath Chakravarthi <[EMAIL PROTECTED]>:
> > > Carp,
> > >  IMHO, .20.x has it. fs.inmemory.size.mb is the reduce-side equivalent
> of
> > io.sort.mb. In the reducer tasks, intermediate map output is collected
> into
> > a buffer (who size is governed by this parameter's value), and data is
> > flushed into files as (partially) sorted KVs.
> > >
> > >  These files will be re-merged if we end up with more than
> io.sort.factor
> > number of files, else KVs will be served out of these files to the reduce
> > function directly.
> > >
> > >  I don't know where in the code it is though, sorry.
> > >
> > > cheers,
> > > Sriguru
> > >
> > >
> > >>-----Original Message-----
> > >>From: Yu Li [mailto:[EMAIL PROTECTED]]
> > >>Sent: Thursday, July 01, 2010 1:12 PM
> > >>To: [EMAIL PROTECTED]
> > >>Subject: In which configuration file to configure the
> > >>"fs.inmemory.size.mb" parameter?
> > >>
> > >>Hi all,
> > >>
> > >>I looked through the "Cluster Setup" guide under link
> > >>http://hadoop.apache.org/common/docs/r0.20.1/cluster_setup.html and
> > >>found there's a "fs.inmemory.size.mb" parameter for specifying memory
> > >>allocated for the in-memory file-system used to merge map-outputs at
> > >>the reduces, and this parameter is set in the "core-site.xml". But
> > >>when I checked the "core-default.xml" under path
> > >>"$HADOOP_HOME/src/core/", I didn't find the parameter at all, nor
> > >>could I find the parameter through JTUI after lauching jobs.
> > >>Does anybody know about this parameter? Has it been removed from
> > >>release 0.20.X? If it hasn't been removed, how could I set the
> > >>parameter besides using the -D option? Thanks in advance.
> > >>
> > >>Best Regards,
> > >>Carp
> > >
> >
>