Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - the last job in the mapreduce plan


Copy link to this message
-
Re: the last job in the mapreduce plan
Ashutosh Chauhan 2010-06-15, 17:24
Gang,

What you are saying can never happen because we create a new MR
operator only when we have a blocking operator which needs to go in
the next MR operator. We dont create new MR operator apriori without
looking at next physical operator in the pipeline. If you are seeing
this happening, I would consider that as a bug.

Ashutosh

On Tue, Jun 15, 2010 at 09:26, Alan Gates <[EMAIL PROTECTED]> wrote:
> I've never seen a case where this happens.  Is this a theoretical question
> or are you seeing this issue?
>
> Alan.
>
> On Jun 15, 2010, at 8:49 AM, Gang Luo wrote:
>
>> Hi,
>> Is it possible the last MapReduce job in the MR plan only loads something
>> and stores it without any other processing in between? For example, when
>> visiting some physical operator, we need to end the current MR operator
>> after embedding the physical operator into MR operator, and create a new MR
>> operator for later physical operators. Unfortunately, the following physical
>> operator is a store, the end of the entire query. In this case, the last MR
>> operator only contain load and store without any meaningful work in between.
>> This idle MapReduce job will degrade the performance. Will this happen in
>> Pig?
>>
>> Thanks,
>> -Gang
>>
>>
>>
>>
>
>