|
|
-
Is there any way for a single map job to show progress
Steve Lewis 2011-10-26, 00:55
My map task needs to handle a large gzipped file and site at 0% forever
until it hits 100%
there is no way to split the file but it would be nice if there were some
indication of progress -
any way to do that or is progress only number of complete tasks?
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
-
Re: Is there any way for a single map job to show progress
Milind.Bhandarkar@... 2011-11-03, 17:28
Individual map task progress is indicative of what percentage of input chunk has been consumed so far by the map task. However, the responsibility of feeding this info to the framework is the responsibility of the record reader.
* Milind
*
From: Steve Lewis <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Tue, 25 Oct 2011 20:55:59 -0400
To: mapreduce-user <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Is there any way for a single map job to show progress
My map task needs to handle a large gzipped file and site at 0% forever until it hits 100%
there is no way to split the file but it would be nice if there were some indication of progress -
any way to do that or is progress only number of complete tasks?
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
-
Re: Is there any way for a single map job to show progress
Steve Lewis 2011-11-03, 18:02
So how does a custom reader to that???
On Thu, Nov 3, 2011 at 10:28 AM, <[EMAIL PROTECTED]> wrote:
> Individual map task progress is indicative of what percentage of input
> chunk has been consumed so far by the map task. However, the responsibility
> of feeding this info to the framework is the responsibility of the record
> reader.
>
>
> * Milind
> *
>
> From: Steve Lewis <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
> Reply-To: "[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>" <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>>
> Date: Tue, 25 Oct 2011 20:55:59 -0400
> To: mapreduce-user <[EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>>
> Subject: Is there any way for a single map job to show progress
>
> My map task needs to handle a large gzipped file and site at 0% forever
> until it hits 100%
> there is no way to split the file but it would be nice if there were some
> indication of progress -
> any way to do that or is progress only number of complete tasks?
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
-
Re: Is there any way for a single map job to show progress
Milind.Bhandarkar@... 2011-11-03, 18:22
By implementing getProgress().
The problem with LineRecordReader is this:
80 if (codec != null) {
81 in = new LineReader(codec.createInputStream(fileIn), job);
82 end = Long.MAX_VALUE;
83 }
And getProgress() is:
Math.min(1.0f, (pos - start) / (float)(end - start));
After the map task finishes, the progress is set to 1.0.
* Milind
*
From: Steve Lewis <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Thu, 3 Nov 2011 14:02:29 -0400
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Is there any way for a single map job to show progress
So how does a custom reader to that???
On Thu, Nov 3, 2011 at 10:28 AM, <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Individual map task progress is indicative of what percentage of input chunk has been consumed so far by the map task. However, the responsibility of feeding this info to the framework is the responsibility of the record reader.
* Milind
*
From: Steve Lewis <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Date: Tue, 25 Oct 2011 20:55:59 -0400
To: mapreduce-user <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]><mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>
Subject: Is there any way for a single map job to show progress
My map task needs to handle a large gzipped file and site at 0% forever until it hits 100%
there is no way to split the file but it would be nice if there were some indication of progress -
any way to do that or is progress only number of complete tasks?
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340<tel:206-384-1340> (cell)
Skype lordjoe_com
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
|
|
