Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> what happens under the hood


+
jamal sasha 2012-12-20, 00:23
Copy link to this message
-
Re: what happens under the hood
This is a very broad question. On the Pig website you can find some papers
on how Pig was implemented, and this should give you a high level view of
what is going on.

For this code, you can use the explain command (explain in; instead of dump
in;) to see the 3 plans that this code generates (logical, physical, mr).
If you want to be a real pro, put in a debug statement in your ide and
actually look at the steps as it builds the logical plan and the converts
to physical and mr.
2012/12/19 jamal sasha <[EMAIL PROTECTED]>

> Hi,
>   I am trying to dig deep on the workings of pig libraries.
>
> So can someone help me understand what happens when someone does:
>
> in = load 'in.txt' using PigStorage(',') as (foo:int);
> dump in;
>
> what happens behind the scenes..
> How does it executes map reduce jobs..
> where is this "load" defined in the pig code base .
> I am just trying to see how  the backend code is implemented where this two
> lines of code translates into the map reduce code.
> Any pointers.
> Thanks
> Jamal
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB