The stack us pretty large and as the job tracker webui shows cleanup completes about 1 sec before the client finishes this does not look like a client issue.

The jobtracker webui shows map

On Sep 27, 2010, at 12:18 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:

> (adding mapreduce-user@ and re-scoping title)
>
> Can you jstack the client while it's waiting 20 seconds? Is it still
> waiting for the job to come back or it's something else? Is the job
> itself done cleaning 20 seconds before the call returns on the client
> side (check the web ui)?
>
> J-D
>
> On Mon, Sep 27, 2010 at 12:10 PM, Pete Tyler <[EMAIL PROTECTED]> wrote:
>> Thanks for the offer, much appreciated I have a very simple mapreduce job on a pseudo distributed system. I have a very small amount of persisted data.
>>
>> Running locally the mapreduce job runs very quickly, less than three seconds.
>>
>> When I run the job against the pseudo distributed hadoop, still on the same machine, as the client then I see the following,
>> - the map and reduce classes run very quickly, a matter of mills in total ... sweet
>> - the client, blocks waiting for the job to finish for about 20 seconds ... very slow
>>
>> I'm trying to understand why I have this 20 second overhead and what I can do about it.
>>
>> My map and reduce classes are in my Hadoop classpath.
>>
>> On Sep 27, 2010, at 11:32 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>>
>>> Using 0.21.0 may reveal newer bugs rather than fixing your older ones.
>>> Maybe we can help you debugging 0.20.2, what are you seeing?
>>>
>>> J-D
>>>

Oops .... Sorry, hit send by mistake.

The stack is pretty large and as the job tracker webui shows cleanup completes about 1 sec before the client finishes this does not look like a client issue. Is that a reasonable assumption?

The jobtracker webui shows map takes 2 secs but reduce from 9 to 12 secs.

Summary:

0 secs: client submits job
+0 secs: jobtracker web ui shows job started
+6 secs: web ui shows map started
+9 secs: web ui shows map complete
+9 secs: web ui shows reduce started
+21 secs: web ui shows reduce complete
+24 secs: web ui shows job cleanup successful

If I change the map class so that it passes on zero records to the reduce, ie reduce input records = 0, the reduce step still takes 9 seconds.

On Sep 27, 2010, at 7:23 PM, Pete Tyler <[EMAIL PROTECTED]> wrote:

> The stack us pretty large and as the job tracker webui shows cleanup completes about 1 sec before the client finishes this does not look like a client issue.
>
> The jobtracker webui shows map

Perhaps the reason of those slowdowns are:
1. copy and unpack job jar.
2. start child java process