Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Understanding Logical Plan


+
Renato Marroquín Mogrovej... 2010-10-24, 19:59
+
Daniel Dai 2010-10-24, 21:13
+
Renato Marroquín Mogrovej... 2010-10-25, 00:37
+
Ashutosh Chauhan 2010-10-24, 23:04
+
Renato Marroquín Mogrovej... 2010-10-29, 02:22
Copy link to this message
-
Re: Understanding Logical Plan
Currently parsing and logical plan building is mixed. So, as script is
getting parsed, its corresponding logical plan
is built. So, at the end of parsing phase, a logical plan is emitted.
Inner plans are also built at the same time. The whole plan
construction is done in once shot. This plan is then run through
validators which do various kind of validations and checks, is then
optimized, then translated into physical plan, then translated into
Map reduce plan, then optimized again and is finally submitted via job
client for execution.

Hope it helps,
Ashutosh

On Thu, Oct 28, 2010 at 19:22, Renato Marroquín Mogrovejo
<[EMAIL PROTECTED]> wrote:
> Thanks Ashutosh! I read the wiki and it was really illustrating (:
> But there is one thing, are all plans (top level and inner plans) made at
> the same time? I mean after parsing the query, or is the top level plan made
> first and then when the executing one of its nodes, then an inner plan is
> made for it?
> Is there anywhere else where I could find any extra information?
>
> Renato M.
>
>
> 2010/10/24 Ashutosh Chauhan <[EMAIL PROTECTED]>
>
>> http://wiki.apache.org/pig/NestedLogicalPlan
>>
>> This doesn't reflect the current implementation but fundamental idea
>> remains the same. You may want to start from there.
>>
>> Ashutosh
>> On Sun, Oct 24, 2010 at 12:59, Renato Marroquín Mogrovejo
>> <[EMAIL PROTECTED]> wrote:
>> > Hi everyone,
>> >
>> > I know Pig builds a logical plan for every bag that the user defines, and
>> > that every new bag will have as its logical plan the combination of the
>> > input bags' logical plans and the comands it uses. Is there any other
>> > command that creates a logical plan? I guess the one that creates the
>> first
>> > logical plan is the STORE comand. And how are the other commands' logical
>> > plans combined? For example if I have a join between two relations, it
>> will
>> > create a single logical plan that contains the loading logical plans, but
>> > how does Pig decide which relation goes where? Or is that a part that
>> > happens while compiling the logical plan into MapReduce and not in the
>> > logical plan?
>> > Thanks in advanced.
>> >
>> > Renato M.
>> >
>>
>
+
Renato Marroquín Mogrovej... 2010-11-03, 02:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB