Pig, mail # user - Re: LOAD once, use multiple times

Re: LOAD once, use multiple times

Jonathan Coveney 2011-10-04, 01:04

If you want to know more about the internals, I'd check out the paper Yahoo
put out on the topic (or, of course, buy the book Programming Pig).

The answer to this is pretty simple: if you load a file multiple times into
different relations, then it will be scanned multiple times. So...

a = load 'thing';
b = load 'thing;

{..stuff using a..}
{..stuff using b..}

would load 'thing' twice. This is done for joins and whatnot -- there are
cases when you need to load the same file separately, twice. What happens is
essentially that you're going to load and scan the data twice.

However, as in your case, if you instead combine the load, then you'd have

a = load 'thing';
{..stuff using a..}
{..stuff using a (which previously used b)..}

Now it will just scan a once, and then go into each of the pipelines you
defined.

Obviously it's more complex than that, but that's the general gist.

2011/10/3 Something Something <[EMAIL PROTECTED]>

> I have 3 Pig scripts that load data from the same log file, but filter &
> group this data differently. If I combine these 3 into one & LOAD only
> once, performance seems to have improved, but now I am curious exactly what
> does LOAD do?
>
> How does LOAD work internally? Does Pig save results of the LOAD into some
> separate location in HDFS? Someone please explain how LOAD relates to
> MapReduce? Thanks.
>

Re: LOAD once, use multiple times

Something Something 2011-10-04, 04:35

Let me ask the question differently. Let's say I was not using Pig. I
wanted to do this just using Java MapReduce. The input file is HUGE. One
obvious way to do this would be to write 3 different MR jobs. But that
means this huge file be read 3 times which is what I am trying to avoid.

Is there a way to write a Mapper that will read this file only once, and
then write to 3 different Reducers with different keys?

Going back to Pig, when I LOAD this file & then later 'group by' 3 different
keys, how does Pig do this? Does it "LOAD" this input file into some
interim file & call 3 different Map Reduce jobs?

If this makes no sense, please ignore it. I will try to use 'Explain',
'Describe' to learn the internals. Thanks.
On Mon, Oct 3, 2011 at 6:04 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:

> If you want to know more about the internals, I'd check out the paper Yahoo
> put out on the topic (or, of course, buy the book Programming Pig).
>
> The answer to this is pretty simple: if you load a file multiple times into
> different relations, then it will be scanned multiple times. So...
>
> a = load 'thing';
> b = load 'thing;
>
> {..stuff using a..}
> {..stuff using b..}
>
> would load 'thing' twice. This is done for joins and whatnot -- there are
> cases when you need to load the same file separately, twice. What happens
> is
> essentially that you're going to load and scan the data twice.
>
> However, as in your case, if you instead combine the load, then you'd have
>
> a = load 'thing';
> {..stuff using a..}
> {..stuff using a (which previously used b)..}
>
> Now it will just scan a once, and then go into each of the pipelines you
> defined.
>
> Obviously it's more complex than that, but that's the general gist.
>
> 2011/10/3 Something Something <[EMAIL PROTECTED]>
>
> > I have 3 Pig scripts that load data from the same log file, but filter &
> > group this data differently. If I combine these 3 into one & LOAD only
> > once, performance seems to have improved, but now I am curious exactly
> what
> > does LOAD do?
> >
> > How does LOAD work internally? Does Pig save results of the LOAD into
> some
> > separate location in HDFS? Someone please explain how LOAD relates to
> > MapReduce? Thanks.
> >
>

Re: LOAD once, use multiple times

Jonathan Coveney 2011-10-04, 15:51

Pig does exactly what you're describing in the second part, but only if you
reuse the same relation. It doesn't load the file into an interim job,
although I'm not a pro at that stage. But in MR, the map output files are
stored to disk, and then the reducers go. In this case, it is smart enough
to have the one map stage, and then the 3 different reducers.
2011/10/3 Something Something <[EMAIL PROTECTED]>

> Let me ask the question differently. Let's say I was not using Pig. I
> wanted to do this just using Java MapReduce. The input file is HUGE. One
> obvious way to do this would be to write 3 different MR jobs. But that
> means this huge file be read 3 times which is what I am trying to avoid.
>
> Is there a way to write a Mapper that will read this file only once, and
> then write to 3 different Reducers with different keys?
>
> Going back to Pig, when I LOAD this file & then later 'group by' 3
> different
> keys, how does Pig do this? Does it "LOAD" this input file into some
> interim file & call 3 different Map Reduce jobs?
>
> If this makes no sense, please ignore it. I will try to use 'Explain',
> 'Describe' to learn the internals. Thanks.
>
>
> On Mon, Oct 3, 2011 at 6:04 PM, Jonathan Coveney <[EMAIL PROTECTED]>
> wrote:
>
> > If you want to know more about the internals, I'd check out the paper
> Yahoo
> > put out on the topic (or, of course, buy the book Programming Pig).
> >
> > The answer to this is pretty simple: if you load a file multiple times
> into
> > different relations, then it will be scanned multiple times. So...
> >
> > a = load 'thing';
> > b = load 'thing;
> >
> > {..stuff using a..}
> > {..stuff using b..}
> >
> > would load 'thing' twice. This is done for joins and whatnot -- there are
> > cases when you need to load the same file separately, twice. What happens
> > is
> > essentially that you're going to load and scan the data twice.
> >
> > However, as in your case, if you instead combine the load, then you'd
> have
> >
> > a = load 'thing';
> > {..stuff using a..}
> > {..stuff using a (which previously used b)..}
> >
> > Now it will just scan a once, and then go into each of the pipelines you
> > defined.
> >
> > Obviously it's more complex than that, but that's the general gist.
> >
> > 2011/10/3 Something Something <[EMAIL PROTECTED]>
> >
> > > I have 3 Pig scripts that load data from the same log file, but filter
> &
> > > group this data differently. If I combine these 3 into one & LOAD only
> > > once, performance seems to have improved, but now I am curious exactly
> > what
> > > does LOAD do?
> > >
> > > How does LOAD work internally? Does Pig save results of the LOAD into
> > some
> > > separate location in HDFS? Someone please explain how LOAD relates to
> > > MapReduce? Thanks.
> > >
> >
>

Re: LOAD once, use multiple times

Thejas Nair 2011-10-04, 17:15

See slides 16,17 in
http://www.slideshare.net/thejasmn/apache-pig-performance-optimizations-talk-at-apachecon-2010
.

For the query in example, pig includes an index in the map output key,
and the reduce has a PODemux operator that sends the records to
appropriate reduce plans. There are no distinct reduce tasks for each
group operation.

-Thejas
On 10/3/11 9:35 PM, Something Something wrote:
> Let me ask the question differently. Let's say I was not using Pig. I
> wanted to do this just using Java MapReduce. The input file is HUGE. One
> obvious way to do this would be to write 3 different MR jobs. But that
> means this huge file be read 3 times which is what I am trying to avoid.
>
> Is there a way to write a Mapper that will read this file only once, and
> then write to 3 different Reducers with different keys?
>
> Going back to Pig, when I LOAD this file& then later 'group by' 3 different
> keys, how does Pig do this? Does it "LOAD" this input file into some
> interim file& call 3 different Map Reduce jobs?
>
> If this makes no sense, please ignore it. I will try to use 'Explain',
> 'Describe' to learn the internals. Thanks.
>
>
> On Mon, Oct 3, 2011 at 6:04 PM, Jonathan Coveney<[EMAIL PROTECTED]> wrote:
>
>> If you want to know more about the internals, I'd check out the paper Yahoo
>> put out on the topic (or, of course, buy the book Programming Pig).
>>
>> The answer to this is pretty simple: if you load a file multiple times into
>> different relations, then it will be scanned multiple times. So...
>>
>> a = load 'thing';
>> b = load 'thing;
>>
>> {..stuff using a..}
>> {..stuff using b..}
>>
>> would load 'thing' twice. This is done for joins and whatnot -- there are
>> cases when you need to load the same file separately, twice. What happens
>> is
>> essentially that you're going to load and scan the data twice.
>>
>> However, as in your case, if you instead combine the load, then you'd have
>>
>> a = load 'thing';
>> {..stuff using a..}
>> {..stuff using a (which previously used b)..}
>>
>> Now it will just scan a once, and then go into each of the pipelines you
>> defined.
>>
>> Obviously it's more complex than that, but that's the general gist.
>>
>> 2011/10/3 Something Something<[EMAIL PROTECTED]>
>>
>>> I have 3 Pig scripts that load data from the same log file, but filter&
>>> group this data differently. If I combine these 3 into one& LOAD only
>>> once, performance seems to have improved, but now I am curious exactly
>> what
>>> does LOAD do?
>>>
>>> How does LOAD work internally? Does Pig save results of the LOAD into
>> some
>>> separate location in HDFS? Someone please explain how LOAD relates to
>>> MapReduce? Thanks.
>>>
>>
>

Re: LOAD once, use multiple times

Something Something 2011-10-04, 23:13

Thanks, Thejas! Great presentation. But the slides 16 & 17 are a bit
different from my example. In your example on slide 16, the output from
filter (B) is used twice. Question is what happens when 'A' (output from
LOAD) is used multiple times.

I ran my pig script with command similar to the following:

pig -e 'explain -dot -out ./my.dot -script myscript.pig' >> explain.txt

I noticed in the Graph that 'LOAD' splits the input into 3 SplitOutput[log]
and then runs 3 MR jobs - which is what I expected. Now I am just trying to
understand how I can create 3 SplitOutput like this outside Pig & feed them
in 3 different MR jobs in Java. Knowing this will make me appreciate Pig
more -:)
On Tue, Oct 4, 2011 at 10:15 AM, Thejas Nair <[EMAIL PROTECTED]> wrote:

> See slides 16,17 in http://www.slideshare.net/**thejasmn/apache-pig-**
> performance-optimizations-**talk-at-apachecon-2010<http://www.slideshare.net/thejasmn/apache-pig-performance-optimizations-talk-at-apachecon-2010>.
>
> For the query in example, pig includes an index in the map output key, and
> the reduce has a PODemux operator that sends the records to appropriate
> reduce plans. There are no distinct reduce tasks for each group operation.
>
> -Thejas
>
>
>
> On 10/3/11 9:35 PM, Something Something wrote:
>
>> Let me ask the question differently. Let's say I was not using Pig. I
>> wanted to do this just using Java MapReduce. The input file is HUGE. One
>> obvious way to do this would be to write 3 different MR jobs. But that
>> means this huge file be read 3 times which is what I am trying to avoid.
>>
>> Is there a way to write a Mapper that will read this file only once, and
>> then write to 3 different Reducers with different keys?
>>
>> Going back to Pig, when I LOAD this file& then later 'group by' 3
>> different
>>
>> keys, how does Pig do this? Does it "LOAD" this input file into some
>> interim file& call 3 different Map Reduce jobs?
>>
>>
>> If this makes no sense, please ignore it. I will try to use 'Explain',
>> 'Describe' to learn the internals. Thanks.
>>
>>
>> On Mon, Oct 3, 2011 at 6:04 PM, Jonathan Coveney<[EMAIL PROTECTED]>
>> wrote:
>>
>> If you want to know more about the internals, I'd check out the paper
>>> Yahoo
>>> put out on the topic (or, of course, buy the book Programming Pig).
>>>
>>> The answer to this is pretty simple: if you load a file multiple times
>>> into
>>> different relations, then it will be scanned multiple times. So...
>>>
>>> a = load 'thing';
>>> b = load 'thing;
>>>
>>> {..stuff using a..}
>>> {..stuff using b..}
>>>
>>> would load 'thing' twice. This is done for joins and whatnot -- there are
>>> cases when you need to load the same file separately, twice. What happens
>>> is
>>> essentially that you're going to load and scan the data twice.
>>>
>>> However, as in your case, if you instead combine the load, then you'd
>>> have
>>>
>>> a = load 'thing';
>>> {..stuff using a..}
>>> {..stuff using a (which previously used b)..}
>>>
>>> Now it will just scan a once, and then go into each of the pipelines you
>>> defined.
>>>
>>> Obviously it's more complex than that, but that's the general gist.
>>>
>>> 2011/10/3 Something Something<mailinglists19@**gmail.com<[EMAIL PROTECTED]>
>>> >
>>>
>>> I have 3 Pig scripts that load data from the same log file, but filter&
>>>> group this data differently. If I combine these 3 into one& LOAD only
>>>>
>>>> once, performance seems to have improved, but now I am curious exactly
>>>>
>>> what
>>>
>>>> does LOAD do?
>>>>
>>>> How does LOAD work internally? Does Pig save results of the LOAD into
>>>>
>>> some
>>>
>>>> separate location in HDFS? Someone please explain how LOAD relates to
>>>> MapReduce? Thanks.
>>>>
>>>>
>>>
>>
>

Re: LOAD once, use multiple times

Thejas Nair 2011-10-04, 23:35

Removing the filter (B) in the example will still result in similar
plan, just that the filter operator will be absent.

For example, the following query uses only 1 MR job. If your explain
plan showed more MR jobs, your query probably has something that does
not allow the work to be done in single MR job.

cat mutli.pig
l = load 'x' as (a,b,c);
g1 = group l by a;
f1 = foreach g1 generate group as a, MAX(l.b);
store f1 into 'out1';

g2 = group l by b;
f2 = foreach g2 generate group as b, MAX(l.c);
store f2 into 'out2';

g3 = group l by c;
f3 = foreach g3 generate group as c, MAX(l.a);
store f3 into 'out3';
I used following explain command. Disabled combiner so that the plan is
smaller (to make it easy to read/understand) -
java -Dpig.exec.nocombiner=true -cp pig.jar org.apache.pig.Main -x
local -e explain -script mutli.pig

Pasted the explain plan output here so that formatting is fine -
http://pastebin.com/yGJs8Q4t

-Thejas

On 10/4/11 4:13 PM, Something Something wrote:
> Thanks, Thejas! Great presentation. But the slides 16& 17 are a bit
> different from my example. In your example on slide 16, the output from
> filter (B) is used twice. Question is what happens when 'A' (output from
> LOAD) is used multiple times.
>
> I ran my pig script with command similar to the following:
>
> pig -e 'explain -dot -out ./my.dot -script myscript.pig'>> explain.txt
>
> I noticed in the Graph that 'LOAD' splits the input into 3 SplitOutput[log]
> and then runs 3 MR jobs - which is what I expected. Now I am just trying to
> understand how I can create 3 SplitOutput like this outside Pig& feed them
> in 3 different MR jobs in Java. Knowing this will make me appreciate Pig
> more -:)
>
>
> On Tue, Oct 4, 2011 at 10:15 AM, Thejas Nair<[EMAIL PROTECTED]> wrote:
>
>> See slides 16,17 in http://www.slideshare.net/**thejasmn/apache-pig-**
>> performance-optimizations-**talk-at-apachecon-2010<http://www.slideshare.net/thejasmn/apache-pig-performance-optimizations-talk-at-apachecon-2010>.
>>
>> For the query in example, pig includes an index in the map output key, and
>> the reduce has a PODemux operator that sends the records to appropriate
>> reduce plans. There are no distinct reduce tasks for each group operation.
>>
>> -Thejas
>>
>>
>>
>> On 10/3/11 9:35 PM, Something Something wrote:
>>
>>> Let me ask the question differently. Let's say I was not using Pig. I
>>> wanted to do this just using Java MapReduce. The input file is HUGE. One
>>> obvious way to do this would be to write 3 different MR jobs. But that
>>> means this huge file be read 3 times which is what I am trying to avoid.
>>>
>>> Is there a way to write a Mapper that will read this file only once, and
>>> then write to 3 different Reducers with different keys?
>>>
>>> Going back to Pig, when I LOAD this file& then later 'group by' 3
>>> different
>>>
>>> keys, how does Pig do this? Does it "LOAD" this input file into some
>>> interim file& call 3 different Map Reduce jobs?
>>>
>>>
>>> If this makes no sense, please ignore it. I will try to use 'Explain',
>>> 'Describe' to learn the internals. Thanks.
>>>
>>>
>>> On Mon, Oct 3, 2011 at 6:04 PM, Jonathan Coveney<[EMAIL PROTECTED]>
>>> wrote:
>>>
>>> If you want to know more about the internals, I'd check out the paper
>>>> Yahoo
>>>> put out on the topic (or, of course, buy the book Programming Pig).
>>>>
>>>> The answer to this is pretty simple: if you load a file multiple times
>>>> into
>>>> different relations, then it will be scanned multiple times. So...
>>>>
>>>> a = load 'thing';
>>>> b = load 'thing;
>>>>
>>>> {..stuff using a..}
>>>> {..stuff using b..}
>>>>
>>>> would load 'thing' twice. This is done for joins and whatnot -- there are
>>>> cases when you need to load the same file separately, twice. What happens
>>>> is
>>>> essentially that you're going to load and scan the data twice.
>>>>
>>>> However, as in your case, if you instead combine the load, then you'd
>>>> have
>