Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - distributed cache


Copy link to this message
-
Re: distributed cache
Yanbo Liang 2012-11-16, 10:28
As far as I know, The local.cache.size parameter controls the size of the
DistributedCache. By default, it’s set to 10 GB.
And the parameter io.sort.mb is not used here, it used as each map task has
a circular memory buffer that it writes the output to.

2012/11/16 yingnan.ma <[EMAIL PROTECTED]>

> **
>
> when I use the distributed cache , I found that when the file is more than 100MB or the number of records are more than 10 million
> ,
>
> the file can not be cache in the memory; and I try to set the io.sort.mb is 200MB ;
> it still can not work, Any suggestion would be fine! Thank you !
>
> 2012-11-16
>
>
>
> Yingnan.Ma
>
> E    [EMAIL PROTECTED]
>
> MSN:  [EMAIL PROTECTED]
>
> QQ: 230624226****
>
> 北京市朝阳区八里庄西里100号东区 住邦2000,1号楼A座2101室,100025****
>
> ********************************************************************  ** *
> *************************************************************************
> 北京・上海・硅谷****
>
> http://www.ipinyou.com****
> ------------------------------
>
>