Hi everyone,

I know Pig builds a logical plan for every bag that the user defines, and
that every new bag will have as its logical plan the combination of the
input bags' logical plans and the comands it uses. Is there any other
command that creates a logical plan? I guess the one that creates the first
logical plan is the STORE comand. And how are the other commands' logical
plans combined? For example if I have a join between two relations, it will
create a single logical plan that contains the loading logical plans, but
how does Pig decide which relation goes where? Or is that a part that
happens while compiling the logical plan into MapReduce and not in the
logical plan?
Thanks in advanced.

Renato M.

Not sure if I understand your question. Logical plan will be created when
you "store", "dump" or "run/exec" a script. Logical plan is merely a
internal structure for your Pig script. If you have script:

A = load 'xxxx';
B = load 'yyyy';
C = join A by $0, B by $0;

Logical plan will be:
load     load
   \         /
      join

We then further transform logical plan to map-reduce plan, which will decide
which map reduce plan each operator goes.

Daniel

-----Original Message-----
From: Renato Marroqu�n Mogrovejo
Sent: Sunday, October 24, 2010 12:59 PM
To: [EMAIL PROTECTED]
Subject: Understanding Logical Plan

Hi everyone,

I know Pig builds a logical plan for every bag that the user defines, and
that every new bag will have as its logical plan the combination of the
input bags' logical plans and the comands it uses. Is there any other
command that creates a logical plan? I guess the one that creates the first
logical plan is the STORE comand. And how are the other commands' logical
plans combined? For example if I have a join between two relations, it will
create a single logical plan that contains the loading logical plans, but
how does Pig decide which relation goes where? Or is that a part that
happens while compiling the logical plan into MapReduce and not in the
logical plan?
Thanks in advanced.

Renato M.

I thought that pig built a logical plan for every bag that the user defines,
and that for next commands it would only combine the previous ones with the
new ones.

2010/10/24 Daniel Dai <[EMAIL PROTECTED]>

> Not sure if I understand your question. Logical plan will be created when
> you "store", "dump" or "run/exec" a script.
>

Don't those commands make the compilation of a logical plan into a physical
one?
I thought it went something like this . . .

Logical plan is merely a internal structure for your Pig script. If you have
> script:
>
> A = load 'xxxx';
>
---> A logical plan is made.

> B = load 'yyyy';
>
---> B logical plan is made.

> C = join A by $0, B by $0;
>
---> C logical plan is made by joining plans A and B which would produce the
logical plan you pointed out

>
> Logical plan will be:
> load     load
>  \         /
>     join
>
>
I thought this was the way it worked because of the paper, have you guys
change it? if so, is there any new documentation on it please? So my
question would be if there are any other commands like the LOAD command that
originate a new logical plan because all the rest will be just combinations
of the previous logical plans (sorry I mistyped in my first email, I was
referening to the LOAD command and not to the STORE one).
Thanks Daniel!
> We then further transform logical plan to map-reduce plan, which will
> decide which map reduce plan each operator goes.
>
> Daniel
>
> -----Original Message----- From: Renato Marroquín Mogrovejo
> Sent: Sunday, October 24, 2010 12:59 PM
> To: [EMAIL PROTECTED]
> Subject: Understanding Logical Plan
>
>
> Hi everyone,
>
> I know Pig builds a logical plan for every bag that the user defines, and
> that every new bag will have as its logical plan the combination of the
> input bags' logical plans and the comands it uses. Is there any other
> command that creates a logical plan? I guess the one that creates the first
> logical plan is the STORE comand. And how are the other commands' logical
> plans combined? For example if I have a join between two relations, it will
> create a single logical plan that contains the loading logical plans, but
> how does Pig decide which relation goes where? Or is that a part that
> happens while compiling the logical plan into MapReduce and not in the
> logical plan?
> Thanks in advanced.
>
> Renato M.
>

http://wiki.apache.org/pig/NestedLogicalPlan

This doesn't reflect the current implementation but fundamental idea
remains the same. You may want to start from there.

Ashutosh
On Sun, Oct 24, 2010 at 12:59, Renato Marroquín Mogrovejo
<[EMAIL PROTECTED]> wrote:
> Hi everyone,
>
> I know Pig builds a logical plan for every bag that the user defines, and
> that every new bag will have as its logical plan the combination of the
> input bags' logical plans and the comands it uses. Is there any other
> command that creates a logical plan? I guess the one that creates the first
> logical plan is the STORE comand. And how are the other commands' logical
> plans combined? For example if I have a join between two relations, it will
> create a single logical plan that contains the loading logical plans, but
> how does Pig decide which relation goes where? Or is that a part that
> happens while compiling the logical plan into MapReduce and not in the
> logical plan?
> Thanks in advanced.
>
> Renato M.
>

Thanks Ashutosh! I read the wiki and it was really illustrating (:
But there is one thing, are all plans (top level and inner plans) made at
the same time? I mean after parsing the query, or is the top level plan made
first and then when the executing one of its nodes, then an inner plan is
made for it?
Is there anywhere else where I could find any extra information?

Renato M.
2010/10/24 Ashutosh Chauhan <[EMAIL PROTECTED]>

> http://wiki.apache.org/pig/NestedLogicalPlan
>
> This doesn't reflect the current implementation but fundamental idea
> remains the same. You may want to start from there.
>
> Ashutosh
> On Sun, Oct 24, 2010 at 12:59, Renato Marroquín Mogrovejo
> <[EMAIL PROTECTED]> wrote:
> > Hi everyone,
> >
> > I know Pig builds a logical plan for every bag that the user defines, and
> > that every new bag will have as its logical plan the combination of the
> > input bags' logical plans and the comands it uses. Is there any other
> > command that creates a logical plan? I guess the one that creates the
> first
> > logical plan is the STORE comand. And how are the other commands' logical
> > plans combined? For example if I have a join between two relations, it
> will
> > create a single logical plan that contains the loading logical plans, but
> > how does Pig decide which relation goes where? Or is that a part that
> > happens while compiling the logical plan into MapReduce and not in the
> > logical plan?
> > Thanks in advanced.
> >
> > Renato M.
> >
>

Currently parsing and logical plan building is mixed. So, as script is
getting parsed, its corresponding logical plan
is built. So, at the end of parsing phase, a logical plan is emitted.
Inner plans are also built at the same time. The whole plan
construction is done in once shot. This plan is then run through
validators which do various kind of validations and checks, is then
optimized, then translated into physical plan, then translated into
Map reduce plan, then optimized again and is finally submitted via job
client for execution.

Hope it helps,
Ashutosh

On Thu, Oct 28, 2010 at 19:22, Renato Marroquín Mogrovejo
<[EMAIL PROTECTED]> wrote:
> Thanks Ashutosh! I read the wiki and it was really illustrating (:
> But there is one thing, are all plans (top level and inner plans) made at
> the same time? I mean after parsing the query, or is the top level plan made
> first and then when the executing one of its nodes, then an inner plan is
> made for it?
> Is there anywhere else where I could find any extra information?
>
> Renato M.
>
>
> 2010/10/24 Ashutosh Chauhan <[EMAIL PROTECTED]>
>
>> http://wiki.apache.org/pig/NestedLogicalPlan
>>
>> This doesn't reflect the current implementation but fundamental idea
>> remains the same. You may want to start from there.
>>
>> Ashutosh
>> On Sun, Oct 24, 2010 at 12:59, Renato Marroquín Mogrovejo
>> <[EMAIL PROTECTED]> wrote:
>> > Hi everyone,
>> >
>> > I know Pig builds a logical plan for every bag that the user defines, and
>> > that every new bag will have as its logical plan the combination of the
>> > input bags' logical plans and the comands it uses. Is there any other
>> > command that creates a logical plan? I guess the one that creates the
>> first
>> > logical plan is the STORE comand. And how are the other commands' logical
>> > plans combined? For example if I have a join between two relations, it
>> will
>> > create a single logical plan that contains the loading logical plans, but
>> > how does Pig decide which relation goes where? Or is that a part that
>> > happens while compiling the logical plan into MapReduce and not in the
>> > logical plan?
>> > Thanks in advanced.
>> >
>> > Renato M.
>> >
>>
>

Thanks Ashutosh! Your explanation and some debugging where quite
enlightening!

Renato M.
2010/10/29 Ashutosh Chauhan <[EMAIL PROTECTED]>

> Currently parsing and logical plan building is mixed. So, as script is
> getting parsed, its corresponding logical plan
> is built. So, at the end of parsing phase, a logical plan is emitted.
> Inner plans are also built at the same time. The whole plan
> construction is done in once shot. This plan is then run through
> validators which do various kind of validations and checks, is then
> optimized, then translated into physical plan, then translated into
> Map reduce plan, then optimized again and is finally submitted via job
> client for execution.
>
> Hope it helps,
> Ashutosh
>
> On Thu, Oct 28, 2010 at 19:22, Renato Marroquín Mogrovejo
> <[EMAIL PROTECTED]> wrote:
> > Thanks Ashutosh! I read the wiki and it was really illustrating (:
> > But there is one thing, are all plans (top level and inner plans) made at
> > the same time? I mean after parsing the query, or is the top level plan
> made
> > first and then when the executing one of its nodes, then an inner plan is
> > made for it?
> > Is there anywhere else where I could find any extra information?
> >
> > Renato M.
> >
> >
> > 2010/10/24 Ashutosh Chauhan <[EMAIL PROTECTED]>
> >
> >> http://wiki.apache.org/pig/NestedLogicalPlan
> >>
> >> This doesn't reflect the current implementation but fundamental idea
> >> remains the same. You may want to start from there.
> >>
> >> Ashutosh
> >> On Sun, Oct 24, 2010 at 12:59, Renato Marroquín Mogrovejo
> >> <[EMAIL PROTECTED]> wrote:
> >> > Hi everyone,
> >> >
> >> > I know Pig builds a logical plan for every bag that the user defines,
> and
> >> > that every new bag will have as its logical plan the combination of
> the
> >> > input bags' logical plans and the comands it uses. Is there any other
> >> > command that creates a logical plan? I guess the one that creates the
> >> first
> >> > logical plan is the STORE comand. And how are the other commands'
> logical
> >> > plans combined? For example if I have a join between two relations, it
> >> will
> >> > create a single logical plan that contains the loading logical plans,
> but
> >> > how does Pig decide which relation goes where? Or is that a part that
> >> > happens while compiling the logical plan into MapReduce and not in the
> >> > logical plan?
> >> > Thanks in advanced.
> >> >
> >> > Renato M.
> >> >
> >>
> >
>