Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Stuck MR job


Copy link to this message
-
Re: Stuck MR job
Hemanth Yamijala 2010-06-23, 17:14
Vidhya,

>>> Can you give the version of Hadoop being used, and configuration of memory related parameters ?
> I did reduce the heap sizes since I wanted to allocate more for Hbase (running in tandem).
> Total mem available: 8 gigs..
>  I allocated 5 gigs for Hbase..
> As it stands, right now:
> MR jobs: 512M each
> 2 max MR jobs
> DN and Task trackers: 1 Gig each
>

Unfortunately, I still need more information:

- What is the version/distribution of Hadoop ? Are you using Capacity
scheduler as the scheduler ?
I would like to know the values set for some specific configuration
variables. Unfortunately, the names of those variables have changed
from 20 to 21 and trunk. Hence, I need to know the version to specify
which ones to look up for.

Thanks
Hemanth

> Vidhya
>
> On 6/23/10 3:16 AM, "Hemanth Yamijala" <[EMAIL PROTECTED]> wrote:
>
> Vidhya,
>
>> Hi
>>  This looks like a trivial problem but would be glad if someone can help..
>>
>>  I have been trying to run a m-r job on my cluster. I had modified my configs (primarily reduced the heap sizes for the task tracker and the data nodes) and restarted my hadoop cluster and the job won't start: The job tracker keeps spewing out logs of the form:
>>
>> 2010-06-23 08:36:28,350 INFO org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker: tracker_b3060080.yst.yahoo.net:localhost/127.0.0.1:55818: Reserved 2 MAP slots for job_201006230709_0008
>> 2010-06-23 08:36:28,350 INFO org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker: tracker_b3060114.yst.yahoo.net:localhost/127.0.0.1:55508: Reserved 2 MAP slots for job_201006230709_0008
>> 2010-06-23 08:36:28,353 INFO org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker: tracker_b3060083.yst.yahoo.net:localhost/127.0.0.1:55675: Reserved 2 MAP slots for job_201006230709_0008
>> 2010-06-23 08:36:28,354 INFO org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker: tracker_b3060054.yst.yahoo.net:localhost/127.0.0.1:59455: Reserved 2 MAP slots for job_201006230709_0008
>> 2010-06-23 08:36:31,352 INFO org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker: tracker_b3060059.yst.yahoo.net:localhost/127.0.0.1:55771: Reserved 2 MAP slots for job_201006230709_0008
>
> This looks like you are using the Capacity scheduler with high memory
> job configuration. Did you set any specific memory parameters ? Also,
> are there no jobs in the cluster other than this one ? Can you give
> the version of Hadoop being used, and configuration of memory related
> parameters ?
>
>>
>> And the cluster shows no activity of the job progressing: (% complete is 0 in the map jobs).. I see no errors/activity in tasktrackers..
>>
>> I would appreciate it if someone helps me with this one.
>> Thank you
>> Vidhya
>>
>
>