Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Pending reducers


+
Evert Lammerts 2012-08-14, 10:39
+
Nitin Pawar 2012-08-14, 10:48
+
Evert Lammerts 2012-08-14, 11:50
Copy link to this message
-
Re: Pending reducers
Nitin Pawar 2012-08-14, 11:56
whats the memory/cpu stats on the machines ? are they exhausted

On Tue, Aug 14, 2012 at 5:20 PM, Evert Lammerts <[EMAIL PROTECTED]> wrote:
>> reducers of multiple jobs do run con-currently as long as they have the
>> resources available.
>
> Yep, and that's what's not happening in my situation. 528 reduce slots, 400 taken by one job, 26 of another job remain in pending state. What could explain this behavior?
>
> Evert
>
>>
>> If you want to limit someone overtaking the cluster, then you can
>> create different job queues and assign quota to each queue. You also
>> have the flexibility of allocating max quota per user in a queue as
>> well.
>>
>>
>>
>> On Tue, Aug 14, 2012 at 4:09 PM, Evert Lammerts
>> <[EMAIL PROTECTED]> wrote:
>> > Hi list,
>> >
>> > I have a cluster running Hadoop 0.20.205 with Kerberos enabled,
>> exposing 528 map slots and 528 reduce slots. Currently somebody is
>> running a NORMAL priority job with 7 mappers and 400 reducers. The
>> mappers have finished and the system is processing the reducers.
>> Another user is running a NORMAL priority job with 1 mapper and 26
>> reducers. The mapper has finished, but the reducers won't come out of
>> "pending" state. There are no other jobs running right now. We've not
>> yet installed a different scheduler, so right now the system is using
>> the default scheduler. How can this behavior be explained? I see
>> mappers of multiple jobs run concurrently, and I *thought* I've seen
>> reducers of multiple jobs run concurrently, but I'm not completely
>> sure. Any idea?
>> >
>> > Evert
>>
>>
>>
>> --
>> Nitin Pawar

--
Nitin Pawar
+
Evert Lammerts 2012-08-14, 12:26
+
Harsh J 2012-08-14, 13:12
+
Evert Lammerts 2012-08-14, 20:45