Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> reducer out of memory?


Copy link to this message
-
Re: reducer out of memory?
thanks, let me run more of this with the settings provided later in
this thread and provide the details

On Wed, May 9, 2012 at 10:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Can you share your job details (or a sample reducer code) and also
> share your exact error?
>
> If you are holding reducer provided values/keys in memory in your
> implementation, it can easily cause an OOME if not handled properly.
> The reducer by itself does read the values off a sorted file on the
> disk and doesn't cache the whole group in memory.
>
> On Thu, May 10, 2012 at 12:20 AM, Yang <[EMAIL PROTECTED]> wrote:
>> it seems that if I put too many records into the same mapper output
>> key, all these records are grouped into one key one one reducer,
>>
>> then the reducer became out of memory.
>>
>>
>> but the reducer interface is:
>>
>>       public void reduce(K key, Iterator<V> values,
>>                          OutputCollector<K, V> output,
>>                          Reporter reporter)
>>
>>
>> so  all the values belonging to the key can be iterated, so
>> theoretically they can be iterated from disk, and does not have to be
>> in memory at the same time,
>> so why am I getting out of heap error? is there some param I could
>> tune (apart from -Xmx since my box is ultimately bounded in memory
>> capacity)
>>
>> thanks
>> Yang
>
>
>
> --
> Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB