Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> 1.2.1 ouput, what is bufvoid


Copy link to this message
-
Re: 1.2.1 ouput, what is bufvoid
The length printed there is the # of elements in an internal array,
not a data size length. If you wish to raise that you can raise up
io.sort.spill.percent (default 0.80, or 80%).

On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <[EMAIL PROTECTED]> wrote:
> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
> only 1mb (1048576)
>
>
> On 08/26/2013 03:14 PM, Harsh J wrote:
>>
>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>
>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <[EMAIL PROTECTED]> wrote:
>>>
>>> I am getting these in job output in the 1.2.1
>>>
>>> Spilling map output: record full = true
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart >>> 111149676; bufend = 122264640; bufvoid = 318767104
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart >>> 1048561; kvend = 838844; length = 1048576
>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>> spill 10
>>>
>>>
>>> my question is what is bufvoid? and how can I increase the amount of the
>>> length?
>>>
>>> thanks,
>>> --
>>> Kaveh Minooie
>>
>>
>>
>>
>
> --
> Kaveh Minooie

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB