Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> number of M/R jobs for a Pig Script


Copy link to this message
-
Re: number of M/R jobs for a Pig Script
And Geert's comment about using external-to-Pig approach reminds me that,
then you have Netflix's PigLipstick too. Nice visual tool for actual
execution and stores job history as well.

Regards,
Shahab
On Tue, Oct 15, 2013 at 8:51 AM, Geert Van Landeghem <[EMAIL PROTECTED]>wrote:

> You can also use ambrose to monitor execution of your pig script at
> runtime. Remark: from pig-0.11 on.
>
> It show you the DAG of MR jobs and which are currently being executed. As
> long as pig-ambrose is connected to the execution of your script (workflow)
> you can replay the workflow.
>
> --
> kind regards,
>  Geert
>
>
>
>
> On 15-okt.-2013, at 14:43, Shahab Yunus <[EMAIL PROTECTED]> wrote:
>
> > Have you tried using ILLUSTRATE and EXPLAIN command? As far as I know, I
> > don't think they give you the exact number as it depends on the actual
> data
> > but I believe you can interpret it/extrapolate it from the information
> > provided by these commands.
> >
> > Regards,
> > Shahab
> >
> >
> > On Tue, Oct 15, 2013 at 3:57 AM, ey-chih chow <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> I have a Pig script that has two group-by statements on the the input
> data
> >> set.  Is there anybody knows how many M-R jobs the script will generate?
> >> Thanks.
> >>
> >> Best regards,
> >>
> >> Ey-Chih Chow
> >>
>
>